2022-03-20T14:56:58.828 INFO:root:teuthology version: 1.1.0-a4e18662 2022-03-20T14:56:58.829 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2022-03-20_14:28:38-rados-wip-yuri6-testing-2022-03-18-1104-distro-default-smithi/6748900 --verbose --owner scheduled_yuriw@teuthology --description rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/calico rook/master} --name yuriw-2022-03-20_14:28:38-rados-wip-yuri6-testing-2022-03-18-1104-distro-default-smithi -- /home/teuthworker/archive/yuriw-2022-03-20_14:28:38-rados-wip-yuri6-testing-2022-03-18-1104-distro-default-smithi/6748900/orig.config.yaml 2022-03-20T14:56:58.861 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-03-20T14:56:58.924 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2022-03-20_14:28:38-rados-wip-yuri6-testing-2022-03-18-1104-distro-default-smithi/6748900 branch: wip-yuri6-testing-2022-03-18-1104 description: rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/calico rook/master} email: null first_in_suite: false job_id: '6748900' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2022-03-20_14:28:38-rados-wip-yuri6-testing-2022-03-18-1104-distro-default-smithi nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri6-testing-2022-03-18-1104 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: e92561bf8a1bc96ca490936c6f46ac4673e1b26e ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: e92561bf8a1bc96ca490936c6f46ac4673e1b26e kernel: hwe: true kubeadm: pod_network: calico version: '1.21' rook: rook_image: rook/ceph:master workunit: branch: wip-yuri6-testing-2022-03-18-1104 sha1: e92561bf8a1bc96ca490936c6f46ac4673e1b26e owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.a - - host.b - client.b - - host.c - client.c sha1: e92561bf8a1bc96ca490936c6f46ac4673e1b26e sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri6-testing-2022-03-18-1104 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: e92561bf8a1bc96ca490936c6f46ac4673e1b26e targets: smithi006.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBOtgprrkF2IqI5/5FqdFoLpEiTM6ShGFIqf6sbSsUMymWiMNEFk/nzABtURrZYAqICuygAYEuSQt+0vgm3nU3NMUukeCmLKl0ypx/kyD530aQ+psyPSnhdoyu688sMekr6pwwyWi6pdjQd60GDouQo3igmbQlyT6UXQvfD8ZPZ/nooNL5rEZhU2duTc7bxxm6+eK06jF/QeGKvWPg1BADhmrl5qIGFtrN7BaHDbOWbg1jjSo5K57dFkuw95VbRQpDQBL2K1Bpv84J58uJcjavYvcJAUz4DFFrkFY2ewngHhQjYXdSNa9p0RLU047jFjtxAhdPwqCE1Bt5nRejtOnAJn6cw+dmGr/4mjWDUojqBQDgIJlsyVWKzJIyLn+VjdiAbBsjK3Vhc4vrZ0TM5eWKN+iOm9dAk++RZJOKHwb/ayx9cqh7O/gaXYrnpTVseY8jTSTAlaoCfsa7igf9j/Dvpdd2DkK5hQ4iMlfX87R104416RwxlEigVEBSkUFECMs= smithi045.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIdL3wh/vJBkxuR+seFaqgaYRY06x363xs10/IwlPZWR8S2WabTalUaYr9oWGvQkrxlRxcO1WFpZRhAMUTxfjGc1p6bMJVc7K5I2SdV+i8EesJ0/hn8OYgR3kAELuKcHLpU1WbIF8tljyO8XLvOvQXWLFM62V18b8b1DBYDqWNoDD36dl48l4mzxO1W4E0JDBJggHXGkXVxT/E4noRTg1RITBQ5dWOqOiFWAiGKlFTIMjyDrtIAMHLCOyG0kzqvZiQluBro7qTyuQ73bBiVWj+vAsXYl9+g4M5cB4JftgAAnbuUWM/jh2sMRFJkjBpAlLqRM7lxR7o4BSgU3VXMEVDeE8BWnMlzduM6VrOMLv7UcTOjV4kXaOSdk9+PABsXOjGbbxH+7xWperMSC7jirEF/K18fcISP5Wm5wUTODqWcmVUKQhql82h93G79lqXfSpC/SFRy3k1quV/0XM9RUcaQVOrGIRVHOx5lH9OnSB+Y/jSgg4iBN+avcHVuNdzrkE= smithi123.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9ZiA87EchXvEMoPg/ezaRMLlwMrU3vYUqInP2WHlGME+lHvH5+prZRV2cuKoxa9QezSdRFNyoMqlVO80et0jGU6HbxVOLoEyvmSRtKTsESLX7Wn2qCyfCx/tyXFCinKx6SdIXh12rer8nK3WT2wK+YY5pz8EwInwhbdJKMLdNru7iXhQ8h0SF7yu8mdyb+XAmVizFKKxxeOtF/Md44Bs41tYjMsEAyY8mfTPV8l3hQNbLDb2YVjhJxMibFipM3NC+VgMAMGQjbezj9DYRKhteKBt2ssOHDOC6BvRMFF0UKuoo5fXUF66kbnudBddfXMquFw2Iyd6kfWPT+AHhBxpmRweOu6v0TjGPLmNzuMsBElnO/DQERqsKxc3wDbsPjN7Hq3NwzN7b0ZoVgHtxiNltyKjzu8SBWsIKbsthUjPJXZ9u2kMy2aGnuMUEU6R/ou8YZWUNH9N88E5G/+8jL59gdnT6NoLscwlutZ01+lylkJHy7Y/C/ynEa5ERKuMtp8c= tasks: - kubeadm: null - nvme_loop: null - rook: null - rook.shell: - ceph -s - ceph orch status - ceph orch ps - ceph orch ls - ceph orch device ls - exec: host.a: - "set -ex\ntoolbox() {\n kubectl -n rook-ceph exec -it deploy/rook-ceph-tools\ \ -- \"$@\"\n}\norig_num_osd=`toolbox ceph osd stat | cut -f3 -d \" \"`\ntoolbox\ \ ceph orch osd rm 0 --force\nremoved_pv=\"\"\nwhile [ \"$removed_pv\" = \"\"\ \ ]\ndo\n removed_pv=`kubectl get pv | grep Released | cut -f1 -d \" \"`\n\ \ sleep 3s\ndone\ntarget_path=`kubectl get pv $removed_pv -o jsonpath='{.spec.local.path}'`\n\ host=`echo $removed_pv | cut -f1 -d \"-\"`\ntoolbox ceph orch device zap $host\ \ $target_path --force\nzap_completion=\"0\"\nwhile [ \"$zap_completion\" =\ \ \"0\" ]\ndo\n zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap\ \ -o jsonpath='{.status.succeeded.path}'`\n sleep 3s\ndone\nkubectl patch\ \ pv $removed_pv -p '{\"spec\":{\"claimRef\": null}}'\ntoolbox ceph orch apply\ \ osd --all-available-devices\nkubectl delete job rook-ceph-device-zap -n rook-ceph\n\ num_osd=\"0\"\nwhile [ \"$num_osd\" != \"$orig_num_osd\" ]\ndo\n echo \"\ waiting for osd to come back up\"\n num_osd=`toolbox ceph osd stat | cut\ \ -f3 -d \" \"`\n sleep 30s\ndone\n" - rook.shell: commands: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch apply rgw foo - ceph orch apply mds foo - ceph orch apply rbd-mirror - ceph orch apply nfs foo --port 12777 teuthology_branch: master teuthology_sha1: a4e18662e5de936eb0ac089e531b933ecbcadb0a timestamp: 2022-03-20_14:28:38 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.22301 2022-03-20T14:56:58.925 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa; will attempt to use it 2022-03-20T14:56:58.926 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks 2022-03-20T14:56:58.926 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-03-20T14:56:58.928 INFO:teuthology.task.internal:Checking packages... 2022-03-20T14:56:58.947 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'e92561bf8a1bc96ca490936c6f46ac4673e1b26e' 2022-03-20T14:56:58.947 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-03-20T14:56:58.948 INFO:teuthology.packaging:ref: None 2022-03-20T14:56:58.949 INFO:teuthology.packaging:tag: None 2022-03-20T14:56:58.949 INFO:teuthology.packaging:branch: wip-yuri6-testing-2022-03-18-1104 2022-03-20T14:56:58.949 INFO:teuthology.packaging:sha1: e92561bf8a1bc96ca490936c6f46ac4673e1b26e 2022-03-20T14:56:58.950 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri6-testing-2022-03-18-1104 2022-03-20T14:56:59.098 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-11187-ge92561bf-1focal 2022-03-20T14:56:59.099 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-03-20T14:56:59.110 INFO:teuthology.task.internal:no buildpackages task found 2022-03-20T14:56:59.110 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-03-20T14:56:59.140 INFO:teuthology.task.internal:Saving configuration 2022-03-20T14:56:59.153 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-03-20T14:56:59.174 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-20T14:56:59.198 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi006.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-03-20_14:28:38-rados-wip-yuri6-testing-2022-03-18-1104-distro-default-smithi/6748900', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-20 14:51:46.034993', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIA76EHtkpUDrFw0CRDscYcs2rOQK1Cmt8Wxg523PnRF5'} 2022-03-20T14:56:59.224 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi045.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-03-20_14:28:38-rados-wip-yuri6-testing-2022-03-18-1104-distro-default-smithi/6748900', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-20 14:51:46.036032', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM7RhKH2Gn7gzdqqzIGRXFiu8Nx2FBXXPzfkDxuk7khC'} 2022-03-20T14:56:59.245 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi123.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-03-20_14:28:38-rados-wip-yuri6-testing-2022-03-18-1104-distro-default-smithi/6748900', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-20 14:51:46.033650', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMIBF2VSspGTfp74+rmyXmkMk0akg5FvtI/SvWic5FqN'} 2022-03-20T14:56:59.246 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-03-20T14:56:59.255 INFO:teuthology.task.internal:roles: ubuntu@smithi006.front.sepia.ceph.com - ['host.a', 'client.a'] 2022-03-20T14:56:59.255 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['host.b', 'client.b'] 2022-03-20T14:56:59.255 INFO:teuthology.task.internal:roles: ubuntu@smithi123.front.sepia.ceph.com - ['host.c', 'client.c'] 2022-03-20T14:56:59.256 INFO:teuthology.run_tasks:Running task console_log... 2022-03-20T14:56:59.383 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6326b65c80>, signals=[15]) 2022-03-20T14:56:59.384 INFO:teuthology.run_tasks:Running task internal.connect... 2022-03-20T14:56:59.395 INFO:teuthology.task.internal:Opening connections... 2022-03-20T14:56:59.396 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi006.front.sepia.ceph.com 2022-03-20T14:56:59.397 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-20T14:56:59.450 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi045.front.sepia.ceph.com 2022-03-20T14:56:59.451 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-20T14:56:59.500 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi123.front.sepia.ceph.com 2022-03-20T14:56:59.500 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-20T14:56:59.552 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-03-20T14:56:59.562 DEBUG:teuthology.orchestra.run.smithi006:> uname -m 2022-03-20T14:56:59.569 INFO:teuthology.orchestra.run.smithi006.stdout:x86_64 2022-03-20T14:56:59.569 DEBUG:teuthology.orchestra.run.smithi006:> cat /etc/os-release 2022-03-20T14:56:59.575 INFO:teuthology.orchestra.run.smithi006.stdout:NAME="Ubuntu" 2022-03-20T14:56:59.576 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-20T14:56:59.576 INFO:teuthology.orchestra.run.smithi006.stdout:ID=ubuntu 2022-03-20T14:56:59.576 INFO:teuthology.orchestra.run.smithi006.stdout:ID_LIKE=debian 2022-03-20T14:56:59.576 INFO:teuthology.orchestra.run.smithi006.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-20T14:56:59.576 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION_ID="20.04" 2022-03-20T14:56:59.577 INFO:teuthology.orchestra.run.smithi006.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-20T14:56:59.577 INFO:teuthology.orchestra.run.smithi006.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-20T14:56:59.577 INFO:teuthology.orchestra.run.smithi006.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-20T14:56:59.577 INFO:teuthology.orchestra.run.smithi006.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-20T14:56:59.578 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION_CODENAME=focal 2022-03-20T14:56:59.578 INFO:teuthology.orchestra.run.smithi006.stdout:UBUNTU_CODENAME=focal 2022-03-20T14:56:59.578 INFO:teuthology.lock.ops:Updating smithi006.front.sepia.ceph.com on lock server 2022-03-20T14:56:59.595 DEBUG:teuthology.orchestra.run.smithi045:> uname -m 2022-03-20T14:56:59.601 INFO:teuthology.orchestra.run.smithi045.stdout:x86_64 2022-03-20T14:56:59.602 DEBUG:teuthology.orchestra.run.smithi045:> cat /etc/os-release 2022-03-20T14:56:59.652 INFO:teuthology.orchestra.run.smithi045.stdout:NAME="Ubuntu" 2022-03-20T14:56:59.652 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-20T14:56:59.652 INFO:teuthology.orchestra.run.smithi045.stdout:ID=ubuntu 2022-03-20T14:56:59.653 INFO:teuthology.orchestra.run.smithi045.stdout:ID_LIKE=debian 2022-03-20T14:56:59.653 INFO:teuthology.orchestra.run.smithi045.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-20T14:56:59.653 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION_ID="20.04" 2022-03-20T14:56:59.653 INFO:teuthology.orchestra.run.smithi045.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-20T14:56:59.653 INFO:teuthology.orchestra.run.smithi045.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-20T14:56:59.654 INFO:teuthology.orchestra.run.smithi045.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-20T14:56:59.654 INFO:teuthology.orchestra.run.smithi045.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-20T14:56:59.654 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION_CODENAME=focal 2022-03-20T14:56:59.654 INFO:teuthology.orchestra.run.smithi045.stdout:UBUNTU_CODENAME=focal 2022-03-20T14:56:59.655 INFO:teuthology.lock.ops:Updating smithi045.front.sepia.ceph.com on lock server 2022-03-20T14:56:59.679 DEBUG:teuthology.orchestra.run.smithi123:> uname -m 2022-03-20T14:56:59.693 INFO:teuthology.orchestra.run.smithi123.stdout:x86_64 2022-03-20T14:56:59.694 DEBUG:teuthology.orchestra.run.smithi123:> cat /etc/os-release 2022-03-20T14:56:59.748 INFO:teuthology.orchestra.run.smithi123.stdout:NAME="Ubuntu" 2022-03-20T14:56:59.749 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-20T14:56:59.749 INFO:teuthology.orchestra.run.smithi123.stdout:ID=ubuntu 2022-03-20T14:56:59.749 INFO:teuthology.orchestra.run.smithi123.stdout:ID_LIKE=debian 2022-03-20T14:56:59.750 INFO:teuthology.orchestra.run.smithi123.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-20T14:56:59.750 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION_ID="20.04" 2022-03-20T14:56:59.751 INFO:teuthology.orchestra.run.smithi123.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-20T14:56:59.751 INFO:teuthology.orchestra.run.smithi123.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-20T14:56:59.752 INFO:teuthology.orchestra.run.smithi123.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-20T14:56:59.753 INFO:teuthology.orchestra.run.smithi123.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-20T14:56:59.753 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION_CODENAME=focal 2022-03-20T14:56:59.753 INFO:teuthology.orchestra.run.smithi123.stdout:UBUNTU_CODENAME=focal 2022-03-20T14:56:59.754 INFO:teuthology.lock.ops:Updating smithi123.front.sepia.ceph.com on lock server 2022-03-20T14:56:59.777 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-03-20T14:56:59.793 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-03-20T14:56:59.823 INFO:teuthology.task.internal:Checking for old test directory... 2022-03-20T14:56:59.823 DEBUG:teuthology.orchestra.run.smithi006:> test '!' -e /home/ubuntu/cephtest 2022-03-20T14:56:59.826 DEBUG:teuthology.orchestra.run.smithi045:> test '!' -e /home/ubuntu/cephtest 2022-03-20T14:56:59.829 DEBUG:teuthology.orchestra.run.smithi123:> test '!' -e /home/ubuntu/cephtest 2022-03-20T14:56:59.835 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-03-20T14:56:59.897 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-03-20T14:56:59.897 DEBUG:teuthology.orchestra.run.smithi006:> test -z $(ls -A /var/lib/ceph) 2022-03-20T14:56:59.900 DEBUG:teuthology.orchestra.run.smithi045:> test -z $(ls -A /var/lib/ceph) 2022-03-20T14:56:59.902 DEBUG:teuthology.orchestra.run.smithi123:> test -z $(ls -A /var/lib/ceph) 2022-03-20T14:56:59.937 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-03-20T14:57:00.011 INFO:teuthology.run_tasks:Running task kernel... 2022-03-20T14:57:00.029 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-03-20T14:57:00.029 INFO:teuthology.task.kernel:normalize config orig: {'hwe': True} 2022-03-20T14:57:00.030 DEBUG:teuthology.task.kernel:normalized overrides {'host.a': {'hwe': True}, 'client.a': {'hwe': True}, 'host.b': {'hwe': True}, 'client.b': {'hwe': True}, 'host.c': {'hwe': True}, 'client.c': {'hwe': True}} 2022-03-20T14:57:00.030 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.b': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.c': {'kdb': True, 'sha1': 'distro', 'hwe': True}}, timeout 300 2022-03-20T14:57:00.030 DEBUG:teuthology.orchestra.run.smithi006:> test -f /run/.containerenv -o -f /.dockerenv 2022-03-20T14:57:00.031 DEBUG:teuthology.orchestra.run.smithi045:> test -f /run/.containerenv -o -f /.dockerenv 2022-03-20T14:57:00.032 DEBUG:teuthology.orchestra.run.smithi123:> test -f /run/.containerenv -o -f /.dockerenv 2022-03-20T14:57:00.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-20T14:57:00.038 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2022-03-20T14:57:00.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-20T14:57:00.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-20T14:57:00.040 DEBUG:teuthology.orchestra.run.smithi123:> uname -r 2022-03-20T14:57:00.041 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2022-03-20T14:57:00.047 INFO:teuthology.orchestra.run.smithi123.stdout:5.4.0-97-generic 2022-03-20T14:57:00.048 INFO:teuthology.task.kernel:Running kernel on smithi123: 5.4.0-97-generic 2022-03-20T14:57:00.048 DEBUG:teuthology.orchestra.run.smithi123:> sudo apt-get clean 2022-03-20T14:57:00.049 INFO:teuthology.orchestra.run.smithi006.stdout:5.4.0-97-generic 2022-03-20T14:57:00.050 INFO:teuthology.task.kernel:Running kernel on smithi006: 5.4.0-97-generic 2022-03-20T14:57:00.050 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-get clean 2022-03-20T14:57:00.087 INFO:teuthology.orchestra.run.smithi045.stdout:5.4.0-97-generic 2022-03-20T14:57:00.087 INFO:teuthology.task.kernel:Running kernel on smithi045: 5.4.0-97-generic 2022-03-20T14:57:00.087 DEBUG:teuthology.orchestra.run.smithi045:> sudo apt-get clean 2022-03-20T14:57:00.158 DEBUG:teuthology.orchestra.run.smithi123:> sudo apt-get update 2022-03-20T14:57:00.169 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-get update 2022-03-20T14:57:00.187 DEBUG:teuthology.orchestra.run.smithi045:> sudo apt-get update 2022-03-20T14:57:00.324 INFO:teuthology.orchestra.run.smithi123.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-20T14:57:00.357 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-20T14:57:00.368 INFO:teuthology.orchestra.run.smithi045.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-20T14:57:00.431 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-20T14:57:00.439 INFO:teuthology.orchestra.run.smithi045.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-20T14:57:00.457 INFO:teuthology.orchestra.run.smithi045.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-20T14:57:00.460 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-20T14:57:00.463 INFO:teuthology.orchestra.run.smithi006.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-20T14:57:00.511 INFO:teuthology.orchestra.run.smithi123.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-20T14:57:00.520 INFO:teuthology.orchestra.run.smithi006.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-20T14:57:00.620 INFO:teuthology.orchestra.run.smithi123.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,317 kB] 2022-03-20T14:57:00.735 INFO:teuthology.orchestra.run.smithi045.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [617 kB] 2022-03-20T14:57:00.752 INFO:teuthology.orchestra.run.smithi123.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [402 kB] 2022-03-20T14:57:00.761 INFO:teuthology.orchestra.run.smithi123.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [231 kB] 2022-03-20T14:57:00.762 INFO:teuthology.orchestra.run.smithi123.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,804 B] 2022-03-20T14:57:00.763 INFO:teuthology.orchestra.run.smithi123.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [799 kB] 2022-03-20T14:57:00.771 INFO:teuthology.orchestra.run.smithi123.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-20T14:57:00.772 INFO:teuthology.orchestra.run.smithi123.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [114 kB] 2022-03-20T14:57:00.773 INFO:teuthology.orchestra.run.smithi123.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-20T14:57:00.773 INFO:teuthology.orchestra.run.smithi123.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [692 kB] 2022-03-20T14:57:00.781 INFO:teuthology.orchestra.run.smithi123.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [546 kB] 2022-03-20T14:57:00.787 INFO:teuthology.orchestra.run.smithi123.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [121 kB] 2022-03-20T14:57:00.788 INFO:teuthology.orchestra.run.smithi123.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.0 kB] 2022-03-20T14:57:00.805 INFO:teuthology.orchestra.run.smithi123.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-20T14:57:00.806 INFO:teuthology.orchestra.run.smithi123.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-20T14:57:00.806 INFO:teuthology.orchestra.run.smithi123.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-20T14:57:00.806 INFO:teuthology.orchestra.run.smithi123.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-20T14:57:00.818 INFO:teuthology.orchestra.run.smithi045.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,642 kB] 2022-03-20T14:57:00.839 INFO:teuthology.orchestra.run.smithi123.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-20T14:57:00.850 INFO:teuthology.orchestra.run.smithi006.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-20T14:57:00.858 INFO:teuthology.orchestra.run.smithi045.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [312 kB] 2022-03-20T14:57:00.858 INFO:teuthology.orchestra.run.smithi045.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-20T14:57:00.859 INFO:teuthology.orchestra.run.smithi045.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-20T14:57:00.859 INFO:teuthology.orchestra.run.smithi045.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [853 kB] 2022-03-20T14:57:00.872 INFO:teuthology.orchestra.run.smithi045.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [122 kB] 2022-03-20T14:57:00.874 INFO:teuthology.orchestra.run.smithi045.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-20T14:57:00.874 INFO:teuthology.orchestra.run.smithi045.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [910 kB] 2022-03-20T14:57:00.884 INFO:teuthology.orchestra.run.smithi045.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [673 kB] 2022-03-20T14:57:00.892 INFO:teuthology.orchestra.run.smithi045.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [202 kB] 2022-03-20T14:57:00.894 INFO:teuthology.orchestra.run.smithi045.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.3 kB] 2022-03-20T14:57:00.905 INFO:teuthology.orchestra.run.smithi045.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.7 kB] 2022-03-20T14:57:00.905 INFO:teuthology.orchestra.run.smithi045.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,432 B] 2022-03-20T14:57:00.906 INFO:teuthology.orchestra.run.smithi045.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-20T14:57:00.907 INFO:teuthology.orchestra.run.smithi045.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-20T14:57:00.907 INFO:teuthology.orchestra.run.smithi045.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-20T14:57:00.907 INFO:teuthology.orchestra.run.smithi045.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-20T14:57:00.925 INFO:teuthology.orchestra.run.smithi045.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-20T14:57:00.925 INFO:teuthology.orchestra.run.smithi045.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-20T14:57:00.926 INFO:teuthology.orchestra.run.smithi045.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-20T14:57:00.926 INFO:teuthology.orchestra.run.smithi045.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-20T14:57:00.931 INFO:teuthology.orchestra.run.smithi006.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [402 kB] 2022-03-20T14:57:00.944 INFO:teuthology.orchestra.run.smithi045.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-20T14:57:00.962 INFO:teuthology.orchestra.run.smithi123.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [617 kB] 2022-03-20T14:57:01.069 INFO:teuthology.orchestra.run.smithi045.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [402 kB] 2022-03-20T14:57:01.076 INFO:teuthology.orchestra.run.smithi006.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [617 kB] 2022-03-20T14:57:01.194 INFO:teuthology.orchestra.run.smithi006.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,317 kB] 2022-03-20T14:57:01.208 INFO:teuthology.orchestra.run.smithi123.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,642 kB] 2022-03-20T14:57:01.318 INFO:teuthology.orchestra.run.smithi045.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,317 kB] 2022-03-20T14:57:01.324 INFO:teuthology.orchestra.run.smithi006.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,642 kB] 2022-03-20T14:57:01.330 INFO:teuthology.orchestra.run.smithi006.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [231 kB] 2022-03-20T14:57:01.348 INFO:teuthology.orchestra.run.smithi123.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [312 kB] 2022-03-20T14:57:01.360 INFO:teuthology.orchestra.run.smithi123.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-20T14:57:01.361 INFO:teuthology.orchestra.run.smithi123.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [853 kB] 2022-03-20T14:57:01.364 INFO:teuthology.orchestra.run.smithi006.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,804 B] 2022-03-20T14:57:01.365 INFO:teuthology.orchestra.run.smithi006.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [799 kB] 2022-03-20T14:57:01.398 INFO:teuthology.orchestra.run.smithi123.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-20T14:57:01.398 INFO:teuthology.orchestra.run.smithi123.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [122 kB] 2022-03-20T14:57:01.400 INFO:teuthology.orchestra.run.smithi006.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-20T14:57:01.401 INFO:teuthology.orchestra.run.smithi006.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [114 kB] 2022-03-20T14:57:01.403 INFO:teuthology.orchestra.run.smithi123.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-20T14:57:01.404 INFO:teuthology.orchestra.run.smithi123.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [910 kB] 2022-03-20T14:57:01.406 INFO:teuthology.orchestra.run.smithi006.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-20T14:57:01.406 INFO:teuthology.orchestra.run.smithi006.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [692 kB] 2022-03-20T14:57:01.432 INFO:teuthology.orchestra.run.smithi123.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [673 kB] 2022-03-20T14:57:01.437 INFO:teuthology.orchestra.run.smithi006.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [546 kB] 2022-03-20T14:57:01.448 INFO:teuthology.orchestra.run.smithi123.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [202 kB] 2022-03-20T14:57:01.451 INFO:teuthology.orchestra.run.smithi123.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.3 kB] 2022-03-20T14:57:01.452 INFO:teuthology.orchestra.run.smithi006.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [121 kB] 2022-03-20T14:57:01.456 INFO:teuthology.orchestra.run.smithi006.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.0 kB] 2022-03-20T14:57:01.462 INFO:teuthology.orchestra.run.smithi006.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [312 kB] 2022-03-20T14:57:01.469 INFO:teuthology.orchestra.run.smithi045.stdout:Get:30 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [231 kB] 2022-03-20T14:57:01.481 INFO:teuthology.orchestra.run.smithi006.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-20T14:57:01.482 INFO:teuthology.orchestra.run.smithi006.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [853 kB] 2022-03-20T14:57:01.484 INFO:teuthology.orchestra.run.smithi123.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,432 B] 2022-03-20T14:57:01.485 INFO:teuthology.orchestra.run.smithi123.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.7 kB] 2022-03-20T14:57:01.486 INFO:teuthology.orchestra.run.smithi123.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-20T14:57:01.486 INFO:teuthology.orchestra.run.smithi123.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-20T14:57:01.486 INFO:teuthology.orchestra.run.smithi123.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-20T14:57:01.488 INFO:teuthology.orchestra.run.smithi045.stdout:Get:31 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,804 B] 2022-03-20T14:57:01.489 INFO:teuthology.orchestra.run.smithi045.stdout:Get:32 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [799 kB] 2022-03-20T14:57:01.494 INFO:teuthology.orchestra.run.smithi006.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-20T14:57:01.494 INFO:teuthology.orchestra.run.smithi006.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-20T14:57:01.495 INFO:teuthology.orchestra.run.smithi006.stdout:Get:24 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-20T14:57:01.495 INFO:teuthology.orchestra.run.smithi006.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-20T14:57:01.508 INFO:teuthology.orchestra.run.smithi123.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-20T14:57:01.509 INFO:teuthology.orchestra.run.smithi006.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-20T14:57:01.510 INFO:teuthology.orchestra.run.smithi006.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [122 kB] 2022-03-20T14:57:01.514 INFO:teuthology.orchestra.run.smithi006.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-20T14:57:01.514 INFO:teuthology.orchestra.run.smithi006.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [910 kB] 2022-03-20T14:57:01.523 INFO:teuthology.orchestra.run.smithi045.stdout:Get:33 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-20T14:57:01.524 INFO:teuthology.orchestra.run.smithi045.stdout:Get:34 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [114 kB] 2022-03-20T14:57:01.528 INFO:teuthology.orchestra.run.smithi045.stdout:Get:35 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-20T14:57:01.528 INFO:teuthology.orchestra.run.smithi045.stdout:Get:36 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [546 kB] 2022-03-20T14:57:01.544 INFO:teuthology.orchestra.run.smithi006.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [673 kB] 2022-03-20T14:57:01.550 INFO:teuthology.orchestra.run.smithi045.stdout:Get:37 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [692 kB] 2022-03-20T14:57:01.563 INFO:teuthology.orchestra.run.smithi006.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [202 kB] 2022-03-20T14:57:01.567 INFO:teuthology.orchestra.run.smithi006.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.3 kB] 2022-03-20T14:57:01.575 INFO:teuthology.orchestra.run.smithi045.stdout:Get:38 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [121 kB] 2022-03-20T14:57:01.579 INFO:teuthology.orchestra.run.smithi045.stdout:Get:39 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.0 kB] 2022-03-20T14:57:01.588 INFO:teuthology.orchestra.run.smithi123.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-20T14:57:01.589 INFO:teuthology.orchestra.run.smithi123.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-20T14:57:01.589 INFO:teuthology.orchestra.run.smithi123.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-20T14:57:01.590 INFO:teuthology.orchestra.run.smithi123.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-20T14:57:01.590 INFO:teuthology.orchestra.run.smithi123.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-20T14:57:01.595 INFO:teuthology.orchestra.run.smithi006.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,432 B] 2022-03-20T14:57:01.596 INFO:teuthology.orchestra.run.smithi006.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.7 kB] 2022-03-20T14:57:01.596 INFO:teuthology.orchestra.run.smithi006.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-20T14:57:01.597 INFO:teuthology.orchestra.run.smithi006.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-20T14:57:01.597 INFO:teuthology.orchestra.run.smithi006.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-20T14:57:01.624 INFO:teuthology.orchestra.run.smithi006.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-20T14:57:01.630 INFO:teuthology.orchestra.run.smithi045.stdout:Get:40 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-20T14:57:01.631 INFO:teuthology.orchestra.run.smithi045.stdout:Get:41 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-20T14:57:01.631 INFO:teuthology.orchestra.run.smithi045.stdout:Get:42 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-20T14:57:01.632 INFO:teuthology.orchestra.run.smithi045.stdout:Get:43 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-20T14:57:01.704 INFO:teuthology.orchestra.run.smithi006.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-20T14:57:01.705 INFO:teuthology.orchestra.run.smithi006.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-20T14:57:01.705 INFO:teuthology.orchestra.run.smithi006.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-20T14:57:01.705 INFO:teuthology.orchestra.run.smithi006.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-20T14:57:01.706 INFO:teuthology.orchestra.run.smithi006.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-20T14:57:03.938 INFO:teuthology.orchestra.run.smithi123.stdout:Fetched 10.2 MB in 2s (6,116 kB/s) 2022-03-20T14:57:04.087 INFO:teuthology.orchestra.run.smithi045.stdout:Fetched 10.2 MB in 2s (5,867 kB/s) 2022-03-20T14:57:04.611 INFO:teuthology.orchestra.run.smithi006.stdout:Fetched 10.2 MB in 2s (4,766 kB/s) 2022-03-20T14:57:05.095 INFO:teuthology.orchestra.run.smithi123.stdout:Reading package lists... 2022-03-20T14:57:05.121 DEBUG:teuthology.orchestra.run.smithi123:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-20T14:57:05.189 INFO:teuthology.orchestra.run.smithi123.stdout:Reading package lists... 2022-03-20T14:57:05.320 INFO:teuthology.orchestra.run.smithi045.stdout:Reading package lists... 2022-03-20T14:57:05.343 DEBUG:teuthology.orchestra.run.smithi045:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-20T14:57:05.345 INFO:teuthology.orchestra.run.smithi123.stdout:Building dependency tree... 2022-03-20T14:57:05.346 INFO:teuthology.orchestra.run.smithi123.stdout:Reading state information... 2022-03-20T14:57:05.410 INFO:teuthology.orchestra.run.smithi045.stdout:Reading package lists... 2022-03-20T14:57:05.477 INFO:teuthology.orchestra.run.smithi123.stdout:The following additional packages will be installed: 2022-03-20T14:57:05.477 INFO:teuthology.orchestra.run.smithi123.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-20T14:57:05.477 INFO:teuthology.orchestra.run.smithi123.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-modules-5.13.0-35-generic 2022-03-20T14:57:05.478 INFO:teuthology.orchestra.run.smithi123.stdout: linux-modules-extra-5.13.0-35-generic thermald upower usbmuxd 2022-03-20T14:57:05.478 INFO:teuthology.orchestra.run.smithi123.stdout:Suggested packages: 2022-03-20T14:57:05.478 INFO:teuthology.orchestra.run.smithi123.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-20T14:57:05.479 INFO:teuthology.orchestra.run.smithi123.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-35-generic 2022-03-20T14:57:05.534 INFO:teuthology.orchestra.run.smithi123.stdout:The following NEW packages will be installed: 2022-03-20T14:57:05.534 INFO:teuthology.orchestra.run.smithi123.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-20T14:57:05.534 INFO:teuthology.orchestra.run.smithi123.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-image-generic-hwe-20.04 2022-03-20T14:57:05.535 INFO:teuthology.orchestra.run.smithi123.stdout: linux-modules-5.13.0-35-generic linux-modules-extra-5.13.0-35-generic 2022-03-20T14:57:05.535 INFO:teuthology.orchestra.run.smithi123.stdout: thermald upower usbmuxd 2022-03-20T14:57:05.588 INFO:teuthology.orchestra.run.smithi045.stdout:Building dependency tree... 2022-03-20T14:57:05.588 INFO:teuthology.orchestra.run.smithi045.stdout:Reading state information... 2022-03-20T14:57:05.591 INFO:teuthology.orchestra.run.smithi123.stdout:0 upgraded, 13 newly installed, 0 to remove and 109 not upgraded. 2022-03-20T14:57:05.591 INFO:teuthology.orchestra.run.smithi123.stdout:Need to get 80.1 MB of archives. 2022-03-20T14:57:05.592 INFO:teuthology.orchestra.run.smithi123.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-20T14:57:05.592 INFO:teuthology.orchestra.run.smithi123.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-20T14:57:05.656 INFO:teuthology.orchestra.run.smithi123.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-20T14:57:05.663 INFO:teuthology.orchestra.run.smithi123.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-20T14:57:05.668 INFO:teuthology.orchestra.run.smithi123.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-20T14:57:05.686 INFO:teuthology.orchestra.run.smithi123.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-20T14:57:05.694 INFO:teuthology.orchestra.run.smithi123.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [18.3 MB] 2022-03-20T14:57:05.741 INFO:teuthology.orchestra.run.smithi045.stdout:The following additional packages will be installed: 2022-03-20T14:57:05.741 INFO:teuthology.orchestra.run.smithi045.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-20T14:57:05.741 INFO:teuthology.orchestra.run.smithi045.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-modules-5.13.0-35-generic 2022-03-20T14:57:05.742 INFO:teuthology.orchestra.run.smithi045.stdout: linux-modules-extra-5.13.0-35-generic thermald upower usbmuxd 2022-03-20T14:57:05.742 INFO:teuthology.orchestra.run.smithi045.stdout:Suggested packages: 2022-03-20T14:57:05.742 INFO:teuthology.orchestra.run.smithi045.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-20T14:57:05.743 INFO:teuthology.orchestra.run.smithi045.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-35-generic 2022-03-20T14:57:05.787 INFO:teuthology.orchestra.run.smithi045.stdout:The following NEW packages will be installed: 2022-03-20T14:57:05.788 INFO:teuthology.orchestra.run.smithi045.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-20T14:57:05.788 INFO:teuthology.orchestra.run.smithi045.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-image-generic-hwe-20.04 2022-03-20T14:57:05.788 INFO:teuthology.orchestra.run.smithi045.stdout: linux-modules-5.13.0-35-generic linux-modules-extra-5.13.0-35-generic 2022-03-20T14:57:05.789 INFO:teuthology.orchestra.run.smithi045.stdout: thermald upower usbmuxd 2022-03-20T14:57:05.847 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2022-03-20T14:57:05.861 INFO:teuthology.orchestra.run.smithi045.stdout:0 upgraded, 13 newly installed, 0 to remove and 109 not upgraded. 2022-03-20T14:57:05.862 INFO:teuthology.orchestra.run.smithi045.stdout:Need to get 80.1 MB of archives. 2022-03-20T14:57:05.862 INFO:teuthology.orchestra.run.smithi045.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-20T14:57:05.862 INFO:teuthology.orchestra.run.smithi045.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-20T14:57:05.871 DEBUG:teuthology.orchestra.run.smithi006:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-20T14:57:05.945 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2022-03-20T14:57:05.946 INFO:teuthology.orchestra.run.smithi045.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-20T14:57:05.956 INFO:teuthology.orchestra.run.smithi045.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-20T14:57:05.957 INFO:teuthology.orchestra.run.smithi045.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-20T14:57:05.981 INFO:teuthology.orchestra.run.smithi123.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [10.0 MB] 2022-03-20T14:57:05.981 INFO:teuthology.orchestra.run.smithi045.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-20T14:57:05.982 INFO:teuthology.orchestra.run.smithi045.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [18.3 MB] 2022-03-20T14:57:06.083 INFO:teuthology.orchestra.run.smithi123.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [51.1 MB] 2022-03-20T14:57:06.123 INFO:teuthology.orchestra.run.smithi006.stdout:Building dependency tree... 2022-03-20T14:57:06.124 INFO:teuthology.orchestra.run.smithi006.stdout:Reading state information... 2022-03-20T14:57:06.280 INFO:teuthology.orchestra.run.smithi006.stdout:The following additional packages will be installed: 2022-03-20T14:57:06.281 INFO:teuthology.orchestra.run.smithi006.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-20T14:57:06.281 INFO:teuthology.orchestra.run.smithi006.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-modules-5.13.0-35-generic 2022-03-20T14:57:06.281 INFO:teuthology.orchestra.run.smithi006.stdout: linux-modules-extra-5.13.0-35-generic thermald upower usbmuxd 2022-03-20T14:57:06.282 INFO:teuthology.orchestra.run.smithi006.stdout:Suggested packages: 2022-03-20T14:57:06.282 INFO:teuthology.orchestra.run.smithi006.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-20T14:57:06.282 INFO:teuthology.orchestra.run.smithi006.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-35-generic 2022-03-20T14:57:06.327 INFO:teuthology.orchestra.run.smithi006.stdout:The following NEW packages will be installed: 2022-03-20T14:57:06.328 INFO:teuthology.orchestra.run.smithi006.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-20T14:57:06.328 INFO:teuthology.orchestra.run.smithi006.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-image-generic-hwe-20.04 2022-03-20T14:57:06.329 INFO:teuthology.orchestra.run.smithi006.stdout: linux-modules-5.13.0-35-generic linux-modules-extra-5.13.0-35-generic 2022-03-20T14:57:06.329 INFO:teuthology.orchestra.run.smithi006.stdout: thermald upower usbmuxd 2022-03-20T14:57:06.507 INFO:teuthology.orchestra.run.smithi006.stdout:0 upgraded, 13 newly installed, 0 to remove and 109 not upgraded. 2022-03-20T14:57:06.507 INFO:teuthology.orchestra.run.smithi006.stdout:Need to get 80.1 MB of archives. 2022-03-20T14:57:06.508 INFO:teuthology.orchestra.run.smithi006.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-20T14:57:06.508 INFO:teuthology.orchestra.run.smithi006.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-20T14:57:06.553 INFO:teuthology.orchestra.run.smithi045.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [10.0 MB] 2022-03-20T14:57:06.717 INFO:teuthology.orchestra.run.smithi045.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [51.1 MB] 2022-03-20T14:57:06.751 INFO:teuthology.orchestra.run.smithi006.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-20T14:57:06.755 INFO:teuthology.orchestra.run.smithi006.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-20T14:57:06.772 INFO:teuthology.orchestra.run.smithi006.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-20T14:57:06.856 INFO:teuthology.orchestra.run.smithi006.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-20T14:57:06.878 INFO:teuthology.orchestra.run.smithi006.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [18.3 MB] 2022-03-20T14:57:07.056 INFO:teuthology.orchestra.run.smithi123.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.35.40~20.04.20 [2,632 B] 2022-03-20T14:57:07.057 INFO:teuthology.orchestra.run.smithi123.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-20T14:57:07.058 INFO:teuthology.orchestra.run.smithi123.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-20T14:57:07.064 INFO:teuthology.orchestra.run.smithi123.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-20T14:57:07.066 INFO:teuthology.orchestra.run.smithi123.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-20T14:57:07.473 INFO:teuthology.orchestra.run.smithi123.stdout:Fetched 80.1 MB in 2s (52.5 MB/s) 2022-03-20T14:57:07.501 INFO:teuthology.orchestra.run.smithi045.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.35.40~20.04.20 [2,632 B] 2022-03-20T14:57:07.501 INFO:teuthology.orchestra.run.smithi045.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-20T14:57:07.502 INFO:teuthology.orchestra.run.smithi045.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-20T14:57:07.508 INFO:teuthology.orchestra.run.smithi045.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-20T14:57:07.510 INFO:teuthology.orchestra.run.smithi045.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-20T14:57:07.658 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-20T14:57:07.689 INFO:teuthology.orchestra.run.smithi006.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [10.0 MB] 2022-03-20T14:57:08.022 INFO:teuthology.orchestra.run.smithi045.stdout:Fetched 80.1 MB in 2s (46.6 MB/s) 2022-03-20T14:57:08.060 INFO:teuthology.orchestra.run.smithi006.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [51.1 MB] 2022-03-20T14:57:08.257 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-20T14:57:09.090 INFO:teuthology.orchestra.run.smithi123.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-20T14:57:09.095 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-20T14:57:09.204 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-20T14:57:09.532 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-20T14:57:09.547 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-20T14:57:09.582 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-20T14:57:09.687 INFO:teuthology.orchestra.run.smithi045.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-20T14:57:09.692 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-20T14:57:09.711 INFO:teuthology.orchestra.run.smithi006.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.35.40~20.04.20 [2,632 B] 2022-03-20T14:57:09.711 INFO:teuthology.orchestra.run.smithi006.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-20T14:57:09.711 INFO:teuthology.orchestra.run.smithi006.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-20T14:57:09.716 INFO:teuthology.orchestra.run.smithi006.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-20T14:57:09.718 INFO:teuthology.orchestra.run.smithi006.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-20T14:57:09.838 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-20T14:57:09.910 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-20T14:57:09.923 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-20T14:57:09.960 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-20T14:57:10.147 INFO:teuthology.orchestra.run.smithi006.stdout:Fetched 80.1 MB in 3s (23.7 MB/s) 2022-03-20T14:57:10.233 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-20T14:57:10.246 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-20T14:57:10.282 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-20T14:57:10.304 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-20T14:57:10.318 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-20T14:57:10.354 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-20T14:57:10.380 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-20T14:57:10.569 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-20T14:57:10.584 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-20T14:57:10.619 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-20T14:57:10.699 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-20T14:57:10.711 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-20T14:57:10.749 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-20T14:57:10.964 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-20T14:57:10.978 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-20T14:57:11.022 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-20T14:57:11.035 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package linux-modules-5.13.0-35-generic. 2022-03-20T14:57:11.047 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../05-linux-modules-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-20T14:57:11.076 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:57:11.367 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-20T14:57:11.381 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-20T14:57:11.425 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-20T14:57:11.711 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package linux-modules-5.13.0-35-generic. 2022-03-20T14:57:11.727 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../05-linux-modules-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-20T14:57:11.753 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:57:11.859 INFO:teuthology.orchestra.run.smithi006.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-20T14:57:11.864 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-20T14:57:12.019 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-20T14:57:12.489 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-20T14:57:12.503 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-20T14:57:12.572 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-20T14:57:12.834 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-20T14:57:12.849 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-20T14:57:12.883 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-20T14:57:13.253 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-20T14:57:13.269 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-20T14:57:13.289 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package linux-image-5.13.0-35-generic. 2022-03-20T14:57:13.302 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../06-linux-image-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-20T14:57:13.311 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-20T14:57:13.401 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:57:13.656 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-20T14:57:13.671 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-20T14:57:13.731 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-20T14:57:13.863 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-35-generic. 2022-03-20T14:57:13.877 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-20T14:57:13.921 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:57:14.034 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package linux-modules-5.13.0-35-generic. 2022-03-20T14:57:14.049 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../05-linux-modules-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-20T14:57:14.063 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package linux-image-5.13.0-35-generic. 2022-03-20T14:57:14.076 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../06-linux-image-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-20T14:57:14.092 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:57:14.174 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:57:14.667 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-35-generic. 2022-03-20T14:57:14.681 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-20T14:57:14.725 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:57:16.377 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package linux-image-5.13.0-35-generic. 2022-03-20T14:57:16.391 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../06-linux-image-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-20T14:57:16.484 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:57:16.988 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-35-generic. 2022-03-20T14:57:17.001 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-20T14:57:17.038 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:57:19.561 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-20T14:57:19.576 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.35.40~20.04.20_amd64.deb ... 2022-03-20T14:57:19.611 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-20T14:57:19.935 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-20T14:57:19.949 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-20T14:57:19.993 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-20T14:57:20.363 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package thermald. 2022-03-20T14:57:20.379 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-20T14:57:20.420 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-20T14:57:20.770 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-20T14:57:20.785 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.35.40~20.04.20_amd64.deb ... 2022-03-20T14:57:20.828 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-20T14:57:21.139 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-20T14:57:21.153 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-20T14:57:21.189 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-20T14:57:21.370 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package upower. 2022-03-20T14:57:21.387 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-20T14:57:21.439 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-20T14:57:21.554 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package thermald. 2022-03-20T14:57:21.567 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-20T14:57:21.623 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-20T14:57:21.837 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package usbmuxd. 2022-03-20T14:57:21.850 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-20T14:57:21.895 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-20T14:57:22.158 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package upower. 2022-03-20T14:57:22.172 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-20T14:57:22.211 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-20T14:57:22.219 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-20T14:57:22.308 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-20T14:57:22.425 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-20T14:57:22.552 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-20T14:57:22.598 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package usbmuxd. 2022-03-20T14:57:22.611 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-20T14:57:22.649 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-20T14:57:22.678 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-20T14:57:22.804 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-20T14:57:22.906 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-20T14:57:22.924 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.35.40~20.04.20_amd64.deb ... 2022-03-20T14:57:22.930 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-20T14:57:22.960 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-20T14:57:22.964 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-20T14:57:23.073 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-20T14:57:23.199 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-20T14:57:23.302 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-20T14:57:23.318 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-20T14:57:23.325 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-20T14:57:23.360 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-20T14:57:23.383 INFO:teuthology.orchestra.run.smithi123.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-20T14:57:23.431 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-20T14:57:23.451 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-20T14:57:23.610 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-20T14:57:23.612 INFO:teuthology.orchestra.run.smithi123.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-20T14:57:23.621 INFO:teuthology.orchestra.run.smithi123.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-20T14:57:23.622 INFO:teuthology.orchestra.run.smithi123.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-20T14:57:23.720 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-20T14:57:23.746 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package thermald. 2022-03-20T14:57:23.761 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-20T14:57:23.805 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-20T14:57:24.115 INFO:teuthology.orchestra.run.smithi123.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-20T14:57:24.162 INFO:teuthology.orchestra.run.smithi045.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-20T14:57:24.246 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-20T14:57:24.349 INFO:teuthology.orchestra.run.smithi123.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-20T14:57:24.354 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package upower. 2022-03-20T14:57:24.370 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-20T14:57:24.394 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-20T14:57:24.418 INFO:teuthology.orchestra.run.smithi045.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-20T14:57:24.422 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-20T14:57:24.424 INFO:teuthology.orchestra.run.smithi045.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-20T14:57:24.425 INFO:teuthology.orchestra.run.smithi045.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-20T14:57:24.705 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-20T14:57:24.705 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-20T14:57:24.871 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package usbmuxd. 2022-03-20T14:57:24.884 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-20T14:57:24.904 INFO:teuthology.orchestra.run.smithi045.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-20T14:57:24.929 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-20T14:57:25.130 INFO:teuthology.orchestra.run.smithi045.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-20T14:57:25.175 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-20T14:57:25.198 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:57:25.253 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-20T14:57:25.401 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-20T14:57:25.493 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-20T14:57:25.493 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-20T14:57:25.527 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-20T14:57:25.653 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-20T14:57:25.779 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-20T14:57:25.922 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-20T14:57:25.963 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:57:26.040 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-20T14:57:26.444 INFO:teuthology.orchestra.run.smithi123.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-35-generic 2022-03-20T14:57:26.445 INFO:teuthology.orchestra.run.smithi123.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-35-generic 2022-03-20T14:57:26.472 INFO:teuthology.orchestra.run.smithi006.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-20T14:57:26.525 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-20T14:57:26.584 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:57:26.697 INFO:teuthology.orchestra.run.smithi006.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-20T14:57:26.705 INFO:teuthology.orchestra.run.smithi006.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-20T14:57:26.705 INFO:teuthology.orchestra.run.smithi006.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-20T14:57:27.208 INFO:teuthology.orchestra.run.smithi006.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-20T14:57:27.255 INFO:teuthology.orchestra.run.smithi045.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-35-generic 2022-03-20T14:57:27.255 INFO:teuthology.orchestra.run.smithi045.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-35-generic 2022-03-20T14:57:27.414 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:57:27.455 INFO:teuthology.orchestra.run.smithi006.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-20T14:57:27.497 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-20T14:57:27.893 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:57:27.898 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-20T14:57:27.898 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-20T14:57:28.539 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:57:28.824 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:57:29.203 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-20T14:57:29.329 INFO:teuthology.orchestra.run.smithi123.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-20T14:57:29.844 INFO:teuthology.orchestra.run.smithi006.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-35-generic 2022-03-20T14:57:29.844 INFO:teuthology.orchestra.run.smithi006.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-35-generic 2022-03-20T14:57:29.995 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:57:30.184 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-20T14:57:30.309 INFO:teuthology.orchestra.run.smithi045.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-20T14:57:31.372 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:57:32.706 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-20T14:57:32.832 INFO:teuthology.orchestra.run.smithi006.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-20T14:57:33.334 INFO:teuthology.orchestra.run.smithi123.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-20T14:57:34.020 INFO:teuthology.orchestra.run.smithi123.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-20T14:57:34.114 INFO:teuthology.orchestra.run.smithi045.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-20T14:57:34.120 INFO:teuthology.orchestra.run.smithi123.stdout:Processing triggers for linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:57:34.308 INFO:teuthology.orchestra.run.smithi123.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-20T14:57:34.308 INFO:teuthology.orchestra.run.smithi123.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-35-generic 2022-03-20T14:57:34.783 INFO:teuthology.orchestra.run.smithi045.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-20T14:57:34.883 INFO:teuthology.orchestra.run.smithi045.stdout:Processing triggers for linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:57:35.077 INFO:teuthology.orchestra.run.smithi045.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-20T14:57:35.077 INFO:teuthology.orchestra.run.smithi045.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-35-generic 2022-03-20T14:57:38.330 INFO:teuthology.orchestra.run.smithi006.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-20T14:57:39.032 INFO:teuthology.orchestra.run.smithi006.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-20T14:57:39.133 INFO:teuthology.orchestra.run.smithi006.stdout:Processing triggers for linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:57:39.323 INFO:teuthology.orchestra.run.smithi006.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-20T14:57:39.323 INFO:teuthology.orchestra.run.smithi006.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-35-generic 2022-03-20T14:57:57.561 INFO:teuthology.orchestra.run.smithi123.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-20T14:57:57.561 INFO:teuthology.orchestra.run.smithi123.stdout:Sourcing file `/etc/default/grub' 2022-03-20T14:57:57.566 INFO:teuthology.orchestra.run.smithi123.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-20T14:57:57.597 INFO:teuthology.orchestra.run.smithi123.stdout:Generating grub configuration file ... 2022-03-20T14:57:57.904 INFO:teuthology.orchestra.run.smithi123.stdout:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-20T14:57:57.927 INFO:teuthology.orchestra.run.smithi123.stdout:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-20T14:57:58.300 INFO:teuthology.orchestra.run.smithi123.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-20T14:57:58.311 INFO:teuthology.orchestra.run.smithi123.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-20T14:57:58.947 INFO:teuthology.orchestra.run.smithi045.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-20T14:57:58.947 INFO:teuthology.orchestra.run.smithi045.stdout:Sourcing file `/etc/default/grub' 2022-03-20T14:57:58.965 INFO:teuthology.orchestra.run.smithi045.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-20T14:57:58.985 INFO:teuthology.orchestra.run.smithi045.stdout:Generating grub configuration file ... 2022-03-20T14:57:59.274 INFO:teuthology.orchestra.run.smithi045.stdout:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-20T14:57:59.308 INFO:teuthology.orchestra.run.smithi045.stdout:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-20T14:57:59.706 INFO:teuthology.orchestra.run.smithi045.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-20T14:57:59.717 INFO:teuthology.orchestra.run.smithi045.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-20T14:58:00.956 INFO:teuthology.orchestra.run.smithi123.stdout:done 2022-03-20T14:58:01.309 DEBUG:teuthology.orchestra.run.smithi123:> dpkg -s linux-image-generic-hwe-20.04 2022-03-20T14:58:01.335 INFO:teuthology.orchestra.run.smithi123.stdout:Package: linux-image-generic-hwe-20.04 2022-03-20T14:58:01.336 INFO:teuthology.orchestra.run.smithi123.stdout:Status: install ok installed 2022-03-20T14:58:01.336 INFO:teuthology.orchestra.run.smithi123.stdout:Priority: optional 2022-03-20T14:58:01.336 INFO:teuthology.orchestra.run.smithi123.stdout:Section: kernel 2022-03-20T14:58:01.336 INFO:teuthology.orchestra.run.smithi123.stdout:Installed-Size: 19 2022-03-20T14:58:01.337 INFO:teuthology.orchestra.run.smithi123.stdout:Maintainer: Ubuntu Kernel Team 2022-03-20T14:58:01.337 INFO:teuthology.orchestra.run.smithi123.stdout:Architecture: amd64 2022-03-20T14:58:01.337 INFO:teuthology.orchestra.run.smithi123.stdout:Source: linux-meta-hwe-5.13 2022-03-20T14:58:01.337 INFO:teuthology.orchestra.run.smithi123.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-20T14:58:01.338 INFO:teuthology.orchestra.run.smithi123.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-20T14:58:01.338 INFO:teuthology.orchestra.run.smithi123.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-20T14:58:01.338 INFO:teuthology.orchestra.run.smithi123.stdout:Recommends: thermald 2022-03-20T14:58:01.338 INFO:teuthology.orchestra.run.smithi123.stdout:Description: Generic Linux kernel image 2022-03-20T14:58:01.338 INFO:teuthology.orchestra.run.smithi123.stdout: This package will always depend on the latest generic kernel image 2022-03-20T14:58:01.339 INFO:teuthology.orchestra.run.smithi123.stdout: available. 2022-03-20T14:58:01.339 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-35-generic 2022-03-20T14:58:01.339 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-20T14:58:01.339 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-20T14:58:01.340 INFO:teuthology.task.kernel:Installing distro kernel on host.c... 2022-03-20T14:58:01.340 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi123.front.sepia.ceph.com, path=None, version=distro) 2022-03-20T14:58:01.340 DEBUG:teuthology.orchestra.run.smithi123:> sudo apt-get clean 2022-03-20T14:58:01.403 DEBUG:teuthology.orchestra.run.smithi123:> sudo apt-get update 2022-03-20T14:58:01.558 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-20T14:58:01.667 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-20T14:58:01.747 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-20T14:58:01.827 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-20T14:58:02.352 INFO:teuthology.orchestra.run.smithi045.stdout:done 2022-03-20T14:58:02.729 DEBUG:teuthology.orchestra.run.smithi045:> dpkg -s linux-image-generic-hwe-20.04 2022-03-20T14:58:02.758 INFO:teuthology.orchestra.run.smithi045.stdout:Package: linux-image-generic-hwe-20.04 2022-03-20T14:58:02.759 INFO:teuthology.orchestra.run.smithi045.stdout:Status: install ok installed 2022-03-20T14:58:02.759 INFO:teuthology.orchestra.run.smithi045.stdout:Priority: optional 2022-03-20T14:58:02.759 INFO:teuthology.orchestra.run.smithi045.stdout:Section: kernel 2022-03-20T14:58:02.760 INFO:teuthology.orchestra.run.smithi045.stdout:Installed-Size: 19 2022-03-20T14:58:02.760 INFO:teuthology.orchestra.run.smithi045.stdout:Maintainer: Ubuntu Kernel Team 2022-03-20T14:58:02.760 INFO:teuthology.orchestra.run.smithi045.stdout:Architecture: amd64 2022-03-20T14:58:02.760 INFO:teuthology.orchestra.run.smithi045.stdout:Source: linux-meta-hwe-5.13 2022-03-20T14:58:02.760 INFO:teuthology.orchestra.run.smithi045.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-20T14:58:02.761 INFO:teuthology.orchestra.run.smithi045.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-20T14:58:02.761 INFO:teuthology.orchestra.run.smithi045.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-20T14:58:02.761 INFO:teuthology.orchestra.run.smithi045.stdout:Recommends: thermald 2022-03-20T14:58:02.761 INFO:teuthology.orchestra.run.smithi045.stdout:Description: Generic Linux kernel image 2022-03-20T14:58:02.762 INFO:teuthology.orchestra.run.smithi045.stdout: This package will always depend on the latest generic kernel image 2022-03-20T14:58:02.762 INFO:teuthology.orchestra.run.smithi045.stdout: available. 2022-03-20T14:58:02.762 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-35-generic 2022-03-20T14:58:02.762 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-20T14:58:02.763 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-20T14:58:02.763 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2022-03-20T14:58:02.763 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi045.front.sepia.ceph.com, path=None, version=distro) 2022-03-20T14:58:02.764 DEBUG:teuthology.orchestra.run.smithi045:> sudo apt-get clean 2022-03-20T14:58:02.835 DEBUG:teuthology.orchestra.run.smithi045:> sudo apt-get update 2022-03-20T14:58:03.093 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-20T14:58:03.094 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-20T14:58:03.095 INFO:teuthology.orchestra.run.smithi123.stdout:Reading package lists... 2022-03-20T14:58:03.118 DEBUG:teuthology.orchestra.run.smithi123:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-20T14:58:03.166 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-20T14:58:03.184 INFO:teuthology.orchestra.run.smithi123.stdout:Reading package lists... 2022-03-20T14:58:03.203 INFO:teuthology.orchestra.run.smithi006.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-20T14:58:03.203 INFO:teuthology.orchestra.run.smithi006.stdout:Sourcing file `/etc/default/grub' 2022-03-20T14:58:03.210 INFO:teuthology.orchestra.run.smithi006.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-20T14:58:03.231 INFO:teuthology.orchestra.run.smithi006.stdout:Generating grub configuration file ... 2022-03-20T14:58:03.241 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-20T14:58:03.355 INFO:teuthology.orchestra.run.smithi123.stdout:Building dependency tree... 2022-03-20T14:58:03.356 INFO:teuthology.orchestra.run.smithi123.stdout:Reading state information... 2022-03-20T14:58:03.541 INFO:teuthology.orchestra.run.smithi123.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.35.40~20.04.20). 2022-03-20T14:58:03.541 INFO:teuthology.orchestra.run.smithi123.stdout:0 upgraded, 0 newly installed, 0 to remove and 109 not upgraded. 2022-03-20T14:58:03.543 DEBUG:teuthology.orchestra.run.smithi123:> dpkg -s linux-image-generic-hwe-20.04 2022-03-20T14:58:03.559 INFO:teuthology.orchestra.run.smithi123.stdout:Package: linux-image-generic-hwe-20.04 2022-03-20T14:58:03.560 INFO:teuthology.orchestra.run.smithi123.stdout:Status: install ok installed 2022-03-20T14:58:03.560 INFO:teuthology.orchestra.run.smithi123.stdout:Priority: optional 2022-03-20T14:58:03.560 INFO:teuthology.orchestra.run.smithi123.stdout:Section: kernel 2022-03-20T14:58:03.560 INFO:teuthology.orchestra.run.smithi123.stdout:Installed-Size: 19 2022-03-20T14:58:03.560 INFO:teuthology.orchestra.run.smithi123.stdout:Maintainer: Ubuntu Kernel Team 2022-03-20T14:58:03.561 INFO:teuthology.orchestra.run.smithi123.stdout:Architecture: amd64 2022-03-20T14:58:03.561 INFO:teuthology.orchestra.run.smithi123.stdout:Source: linux-meta-hwe-5.13 2022-03-20T14:58:03.561 INFO:teuthology.orchestra.run.smithi123.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-20T14:58:03.561 INFO:teuthology.orchestra.run.smithi123.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-20T14:58:03.562 INFO:teuthology.orchestra.run.smithi123.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-20T14:58:03.562 INFO:teuthology.orchestra.run.smithi123.stdout:Recommends: thermald 2022-03-20T14:58:03.562 INFO:teuthology.orchestra.run.smithi123.stdout:Description: Generic Linux kernel image 2022-03-20T14:58:03.562 INFO:teuthology.orchestra.run.smithi123.stdout: This package will always depend on the latest generic kernel image 2022-03-20T14:58:03.562 INFO:teuthology.orchestra.run.smithi123.stdout: available. 2022-03-20T14:58:03.564 DEBUG:teuthology.orchestra.run.smithi123:> mktemp 2022-03-20T14:58:03.570 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.EWTB2SGn4G 2022-03-20T14:58:03.571 DEBUG:teuthology.orchestra.run.smithi123:> sudo cp /boot/grub/grub.cfg /tmp/tmp.EWTB2SGn4G 2022-03-20T14:58:03.628 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.EWTB2SGn4G 2022-03-20T14:58:03.629 INFO:teuthology.orchestra.run.smithi006.stdout:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-20T14:58:03.669 INFO:teuthology.orchestra.run.smithi006.stdout:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-20T14:58:03.703 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.EWTB2SGn4G is 8KB 2022-03-20T14:58:03.720 DEBUG:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.EWTB2SGn4G 2022-03-20T14:58:03.728 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-20T14:58:03.791 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2022-03-20T14:58:03.792 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-20T14:58:03.792 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-20T14:58:03.828 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-35-generic 2022-03-20T14:58:03.828 DEBUG:teuthology.orchestra.run.smithi123:> sudo update-grub 2022-03-20T14:58:04.123 INFO:teuthology.orchestra.run.smithi006.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-20T14:58:04.134 INFO:teuthology.orchestra.run.smithi006.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-20T14:58:04.679 INFO:teuthology.orchestra.run.smithi045.stdout:Reading package lists... 2022-03-20T14:58:04.704 DEBUG:teuthology.orchestra.run.smithi045:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-20T14:58:04.775 INFO:teuthology.orchestra.run.smithi045.stdout:Reading package lists... 2022-03-20T14:58:04.952 INFO:teuthology.orchestra.run.smithi045.stdout:Building dependency tree... 2022-03-20T14:58:04.953 INFO:teuthology.orchestra.run.smithi123.stderr:Sourcing file `/etc/default/grub' 2022-03-20T14:58:04.953 INFO:teuthology.orchestra.run.smithi045.stdout:Reading state information... 2022-03-20T14:58:04.954 INFO:teuthology.orchestra.run.smithi123.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-20T14:58:04.963 INFO:teuthology.orchestra.run.smithi123.stderr:Generating grub configuration file ... 2022-03-20T14:58:05.135 INFO:teuthology.orchestra.run.smithi045.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.35.40~20.04.20). 2022-03-20T14:58:05.135 INFO:teuthology.orchestra.run.smithi045.stdout:0 upgraded, 0 newly installed, 0 to remove and 109 not upgraded. 2022-03-20T14:58:05.137 DEBUG:teuthology.orchestra.run.smithi045:> dpkg -s linux-image-generic-hwe-20.04 2022-03-20T14:58:05.152 INFO:teuthology.orchestra.run.smithi045.stdout:Package: linux-image-generic-hwe-20.04 2022-03-20T14:58:05.152 INFO:teuthology.orchestra.run.smithi045.stdout:Status: install ok installed 2022-03-20T14:58:05.152 INFO:teuthology.orchestra.run.smithi045.stdout:Priority: optional 2022-03-20T14:58:05.153 INFO:teuthology.orchestra.run.smithi045.stdout:Section: kernel 2022-03-20T14:58:05.153 INFO:teuthology.orchestra.run.smithi045.stdout:Installed-Size: 19 2022-03-20T14:58:05.153 INFO:teuthology.orchestra.run.smithi045.stdout:Maintainer: Ubuntu Kernel Team 2022-03-20T14:58:05.153 INFO:teuthology.orchestra.run.smithi045.stdout:Architecture: amd64 2022-03-20T14:58:05.154 INFO:teuthology.orchestra.run.smithi045.stdout:Source: linux-meta-hwe-5.13 2022-03-20T14:58:05.154 INFO:teuthology.orchestra.run.smithi045.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-20T14:58:05.154 INFO:teuthology.orchestra.run.smithi045.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-20T14:58:05.154 INFO:teuthology.orchestra.run.smithi045.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-20T14:58:05.155 INFO:teuthology.orchestra.run.smithi045.stdout:Recommends: thermald 2022-03-20T14:58:05.155 INFO:teuthology.orchestra.run.smithi045.stdout:Description: Generic Linux kernel image 2022-03-20T14:58:05.155 INFO:teuthology.orchestra.run.smithi045.stdout: This package will always depend on the latest generic kernel image 2022-03-20T14:58:05.156 INFO:teuthology.orchestra.run.smithi045.stdout: available. 2022-03-20T14:58:05.157 DEBUG:teuthology.orchestra.run.smithi045:> mktemp 2022-03-20T14:58:05.204 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/tmp.NDNCqXTch3 2022-03-20T14:58:05.205 DEBUG:teuthology.orchestra.run.smithi045:> sudo cp /boot/grub/grub.cfg /tmp/tmp.NDNCqXTch3 2022-03-20T14:58:05.206 INFO:teuthology.orchestra.run.smithi123.stderr:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-20T14:58:05.218 INFO:teuthology.orchestra.run.smithi123.stderr:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-20T14:58:05.220 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0666 /tmp/tmp.NDNCqXTch3 2022-03-20T14:58:05.257 DEBUG:teuthology.orchestra.remote:smithi045:/tmp/tmp.NDNCqXTch3 is 8KB 2022-03-20T14:58:05.269 DEBUG:teuthology.orchestra.run.smithi045:> rm -fr /tmp/tmp.NDNCqXTch3 2022-03-20T14:58:05.276 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-20T14:58:05.334 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2022-03-20T14:58:05.335 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-20T14:58:05.335 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-20T14:58:05.387 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-35-generic 2022-03-20T14:58:05.388 DEBUG:teuthology.orchestra.run.smithi045:> sudo update-grub 2022-03-20T14:58:05.577 INFO:teuthology.orchestra.run.smithi123.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-20T14:58:05.589 INFO:teuthology.orchestra.run.smithi123.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-20T14:58:06.522 INFO:teuthology.orchestra.run.smithi045.stderr:Sourcing file `/etc/default/grub' 2022-03-20T14:58:06.530 INFO:teuthology.orchestra.run.smithi045.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-20T14:58:06.550 INFO:teuthology.orchestra.run.smithi045.stderr:Generating grub configuration file ... 2022-03-20T14:58:06.785 INFO:teuthology.orchestra.run.smithi045.stderr:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-20T14:58:06.798 INFO:teuthology.orchestra.run.smithi045.stderr:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-20T14:58:06.832 INFO:teuthology.orchestra.run.smithi006.stdout:done 2022-03-20T14:58:07.142 INFO:teuthology.orchestra.run.smithi045.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-20T14:58:07.154 INFO:teuthology.orchestra.run.smithi045.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-20T14:58:07.176 DEBUG:teuthology.orchestra.run.smithi006:> dpkg -s linux-image-generic-hwe-20.04 2022-03-20T14:58:07.186 INFO:teuthology.orchestra.run.smithi123.stderr:done 2022-03-20T14:58:07.188 DEBUG:teuthology.orchestra.run.smithi123:> sudo shutdown -r now 2022-03-20T14:58:07.203 INFO:teuthology.orchestra.run.smithi006.stdout:Package: linux-image-generic-hwe-20.04 2022-03-20T14:58:07.203 INFO:teuthology.orchestra.run.smithi006.stdout:Status: install ok installed 2022-03-20T14:58:07.204 INFO:teuthology.orchestra.run.smithi006.stdout:Priority: optional 2022-03-20T14:58:07.204 INFO:teuthology.orchestra.run.smithi006.stdout:Section: kernel 2022-03-20T14:58:07.204 INFO:teuthology.orchestra.run.smithi006.stdout:Installed-Size: 19 2022-03-20T14:58:07.204 INFO:teuthology.orchestra.run.smithi006.stdout:Maintainer: Ubuntu Kernel Team 2022-03-20T14:58:07.205 INFO:teuthology.orchestra.run.smithi006.stdout:Architecture: amd64 2022-03-20T14:58:07.205 INFO:teuthology.orchestra.run.smithi006.stdout:Source: linux-meta-hwe-5.13 2022-03-20T14:58:07.205 INFO:teuthology.orchestra.run.smithi006.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-20T14:58:07.205 INFO:teuthology.orchestra.run.smithi006.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-20T14:58:07.206 INFO:teuthology.orchestra.run.smithi006.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-20T14:58:07.206 INFO:teuthology.orchestra.run.smithi006.stdout:Recommends: thermald 2022-03-20T14:58:07.206 INFO:teuthology.orchestra.run.smithi006.stdout:Description: Generic Linux kernel image 2022-03-20T14:58:07.206 INFO:teuthology.orchestra.run.smithi006.stdout: This package will always depend on the latest generic kernel image 2022-03-20T14:58:07.207 INFO:teuthology.orchestra.run.smithi006.stdout: available. 2022-03-20T14:58:07.207 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-35-generic 2022-03-20T14:58:07.207 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-20T14:58:07.207 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-20T14:58:07.208 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2022-03-20T14:58:07.208 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi006.front.sepia.ceph.com, path=None, version=distro) 2022-03-20T14:58:07.208 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-get clean 2022-03-20T14:58:07.274 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-get update 2022-03-20T14:58:07.530 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-20T14:58:07.530 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-20T14:58:07.599 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-20T14:58:07.683 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-20T14:58:08.697 INFO:teuthology.orchestra.run.smithi045.stderr:done 2022-03-20T14:58:08.699 DEBUG:teuthology.orchestra.run.smithi045:> sudo shutdown -r now 2022-03-20T14:58:09.094 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2022-03-20T14:58:09.117 DEBUG:teuthology.orchestra.run.smithi006:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-20T14:58:09.182 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2022-03-20T14:58:09.359 INFO:teuthology.orchestra.run.smithi006.stdout:Building dependency tree... 2022-03-20T14:58:09.359 INFO:teuthology.orchestra.run.smithi006.stdout:Reading state information... 2022-03-20T14:58:09.556 INFO:teuthology.orchestra.run.smithi006.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.35.40~20.04.20). 2022-03-20T14:58:09.557 INFO:teuthology.orchestra.run.smithi006.stdout:0 upgraded, 0 newly installed, 0 to remove and 109 not upgraded. 2022-03-20T14:58:09.559 DEBUG:teuthology.orchestra.run.smithi006:> dpkg -s linux-image-generic-hwe-20.04 2022-03-20T14:58:09.578 INFO:teuthology.orchestra.run.smithi006.stdout:Package: linux-image-generic-hwe-20.04 2022-03-20T14:58:09.578 INFO:teuthology.orchestra.run.smithi006.stdout:Status: install ok installed 2022-03-20T14:58:09.578 INFO:teuthology.orchestra.run.smithi006.stdout:Priority: optional 2022-03-20T14:58:09.579 INFO:teuthology.orchestra.run.smithi006.stdout:Section: kernel 2022-03-20T14:58:09.579 INFO:teuthology.orchestra.run.smithi006.stdout:Installed-Size: 19 2022-03-20T14:58:09.579 INFO:teuthology.orchestra.run.smithi006.stdout:Maintainer: Ubuntu Kernel Team 2022-03-20T14:58:09.579 INFO:teuthology.orchestra.run.smithi006.stdout:Architecture: amd64 2022-03-20T14:58:09.580 INFO:teuthology.orchestra.run.smithi006.stdout:Source: linux-meta-hwe-5.13 2022-03-20T14:58:09.580 INFO:teuthology.orchestra.run.smithi006.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-20T14:58:09.580 INFO:teuthology.orchestra.run.smithi006.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-20T14:58:09.580 INFO:teuthology.orchestra.run.smithi006.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-20T14:58:09.581 INFO:teuthology.orchestra.run.smithi006.stdout:Recommends: thermald 2022-03-20T14:58:09.581 INFO:teuthology.orchestra.run.smithi006.stdout:Description: Generic Linux kernel image 2022-03-20T14:58:09.581 INFO:teuthology.orchestra.run.smithi006.stdout: This package will always depend on the latest generic kernel image 2022-03-20T14:58:09.581 INFO:teuthology.orchestra.run.smithi006.stdout: available. 2022-03-20T14:58:09.582 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2022-03-20T14:58:09.588 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.uEI4mZ4Jma 2022-03-20T14:58:09.589 DEBUG:teuthology.orchestra.run.smithi006:> sudo cp /boot/grub/grub.cfg /tmp/tmp.uEI4mZ4Jma 2022-03-20T14:58:09.646 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.uEI4mZ4Jma 2022-03-20T14:58:09.683 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.uEI4mZ4Jma is 8KB 2022-03-20T14:58:09.696 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.uEI4mZ4Jma 2022-03-20T14:58:09.703 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-20T14:58:09.761 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2022-03-20T14:58:09.762 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-20T14:58:09.762 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-20T14:58:09.793 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-35-generic 2022-03-20T14:58:09.794 DEBUG:teuthology.orchestra.run.smithi006:> sudo update-grub 2022-03-20T14:58:10.939 INFO:teuthology.orchestra.run.smithi006.stderr:Sourcing file `/etc/default/grub' 2022-03-20T14:58:10.940 INFO:teuthology.orchestra.run.smithi006.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-20T14:58:10.947 INFO:teuthology.orchestra.run.smithi006.stderr:Generating grub configuration file ... 2022-03-20T14:58:11.253 INFO:teuthology.orchestra.run.smithi006.stderr:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-20T14:58:11.265 INFO:teuthology.orchestra.run.smithi006.stderr:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-20T14:58:11.672 INFO:teuthology.orchestra.run.smithi006.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-20T14:58:11.683 INFO:teuthology.orchestra.run.smithi006.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-20T14:58:13.320 INFO:teuthology.orchestra.run.smithi006.stderr:done 2022-03-20T14:58:13.322 DEBUG:teuthology.orchestra.run.smithi006:> sudo shutdown -r now 2022-03-20T14:58:37.196 INFO:teuthology.task.kernel:Checking client host.c for new kernel version... 2022-03-20T14:58:37.197 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2022-03-20T14:58:37.199 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-20T14:58:38.703 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2022-03-20T14:58:38.704 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2022-03-20T14:58:38.704 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-20T14:58:43.329 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2022-03-20T14:58:43.330 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2022-03-20T14:58:43.331 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-20T14:58:55.567 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.123 2022-03-20T14:58:57.098 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.45 2022-03-20T14:59:01.706 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.6 2022-03-20T14:59:25.591 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2022-03-20T14:59:25.592 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-20T14:59:27.102 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2022-03-20T14:59:27.103 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-20T14:59:31.712 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2022-03-20T14:59:31.713 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-20T14:59:41.899 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.123 2022-03-20T14:59:42.829 DEBUG:teuthology.orchestra.run.smithi045:> true 2022-03-20T14:59:43.603 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi045.front.sepia.ceph.com' 2022-03-20T14:59:43.604 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.13.0-35-generic"... 2022-03-20T14:59:43.604 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2022-03-20T14:59:43.626 INFO:teuthology.orchestra.run.smithi045.stdout:5.13.0-35-generic 2022-03-20T14:59:43.627 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-35-generic vs 5.13.0-35-generic 2022-03-20T14:59:43.627 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-20T14:59:43.627 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-20T14:59:44.628 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2022-03-20T14:59:44.629 DEBUG:teuthology.orchestra.run.smithi045:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-20T14:59:44.699 INFO:teuthology.orchestra.run.smithi045.stdout:ttyS1 2022-03-20T14:59:44.741 DEBUG:teuthology.parallel:result is None 2022-03-20T14:59:47.325 DEBUG:teuthology.orchestra.run.smithi006:> true 2022-03-20T14:59:48.058 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2022-03-20T14:59:48.058 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.13.0-35-generic"... 2022-03-20T14:59:48.059 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2022-03-20T14:59:48.063 INFO:teuthology.orchestra.run.smithi006.stdout:5.13.0-35-generic 2022-03-20T14:59:48.064 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-35-generic vs 5.13.0-35-generic 2022-03-20T14:59:48.064 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-20T14:59:48.064 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-20T14:59:49.065 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2022-03-20T14:59:49.065 DEBUG:teuthology.orchestra.run.smithi006:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-20T14:59:49.218 INFO:teuthology.orchestra.run.smithi006.stdout:ttyS1 2022-03-20T14:59:49.250 DEBUG:teuthology.parallel:result is None 2022-03-20T15:00:11.900 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2022-03-20T15:00:11.902 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-20T15:00:12.077 DEBUG:teuthology.orchestra.run.smithi123:> true 2022-03-20T15:00:12.587 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi123.front.sepia.ceph.com' 2022-03-20T15:00:12.588 INFO:teuthology.task.kernel:Checking kernel version of host.c, want "5.13.0-35-generic"... 2022-03-20T15:00:12.588 DEBUG:teuthology.orchestra.run.smithi123:> uname -r 2022-03-20T15:00:12.593 INFO:teuthology.orchestra.run.smithi123.stdout:5.13.0-35-generic 2022-03-20T15:00:12.593 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-35-generic vs 5.13.0-35-generic 2022-03-20T15:00:12.594 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-20T15:00:12.594 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-20T15:00:13.595 INFO:teuthology.task.kernel:Enabling kdb on host.c... 2022-03-20T15:00:13.595 DEBUG:teuthology.orchestra.run.smithi123:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-20T15:00:13.659 INFO:teuthology.orchestra.run.smithi123.stdout:ttyS1 2022-03-20T15:00:13.690 DEBUG:teuthology.parallel:result is None 2022-03-20T15:00:13.690 INFO:teuthology.run_tasks:Running task internal.base... 2022-03-20T15:00:13.705 INFO:teuthology.task.internal:Creating test directory... 2022-03-20T15:00:13.706 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-20T15:00:13.715 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-20T15:00:13.722 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-20T15:00:13.735 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-03-20T15:00:13.799 INFO:teuthology.run_tasks:Running task internal.archive... 2022-03-20T15:00:13.862 INFO:teuthology.task.internal:Creating archive directory... 2022-03-20T15:00:13.863 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-20T15:00:13.869 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-20T15:00:13.874 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-20T15:00:13.911 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-03-20T15:00:13.924 INFO:teuthology.task.internal:Enabling coredump saving... 2022-03-20T15:00:13.925 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-20T15:00:13.932 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-20T15:00:13.937 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-20T15:00:13.953 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-20T15:00:13.955 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-20T15:00:13.961 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-20T15:00:13.965 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-20T15:00:13.987 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-20T15:00:13.998 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-20T15:00:14.000 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-03-20T15:00:14.016 INFO:teuthology.task.internal:Configuring sudo... 2022-03-20T15:00:14.016 DEBUG:teuthology.orchestra.run.smithi006:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-20T15:00:14.024 DEBUG:teuthology.orchestra.run.smithi045:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-20T15:00:14.032 DEBUG:teuthology.orchestra.run.smithi123:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-20T15:00:14.081 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-03-20T15:00:14.160 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-03-20T15:00:14.161 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-20T15:00:14.169 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-20T15:00:14.175 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-20T15:00:14.191 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-20T15:00:14.233 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-20T15:00:14.287 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2022-03-20T15:00:14.287 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-20T15:00:14.369 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-20T15:00:14.380 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-20T15:00:14.437 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2022-03-20T15:00:14.438 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-20T15:00:14.507 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-20T15:00:14.522 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-20T15:00:14.582 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2022-03-20T15:00:14.582 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-20T15:00:14.665 DEBUG:teuthology.orchestra.run.smithi006:> sudo service rsyslog restart 2022-03-20T15:00:14.669 DEBUG:teuthology.orchestra.run.smithi045:> sudo service rsyslog restart 2022-03-20T15:00:14.673 DEBUG:teuthology.orchestra.run.smithi123:> sudo service rsyslog restart 2022-03-20T15:00:14.764 INFO:teuthology.run_tasks:Running task internal.timer... 2022-03-20T15:00:14.774 INFO:teuthology.task.internal:Starting timer... 2022-03-20T15:00:14.774 INFO:teuthology.run_tasks:Running task pcp... 2022-03-20T15:00:14.806 INFO:teuthology.run_tasks:Running task selinux... 2022-03-20T15:00:14.840 DEBUG:teuthology.task.selinux:Excluding smithi006: OS 'ubuntu' does not support SELinux 2022-03-20T15:00:14.840 DEBUG:teuthology.task.selinux:Excluding smithi045: OS 'ubuntu' does not support SELinux 2022-03-20T15:00:14.841 DEBUG:teuthology.task.selinux:Excluding smithi123: OS 'ubuntu' does not support SELinux 2022-03-20T15:00:14.841 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-03-20T15:00:14.841 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-03-20T15:00:14.842 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-03-20T15:00:14.842 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-03-20T15:00:14.901 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2022-03-20T15:00:14.908 INFO:teuthology.repo_utils:Fetching master from origin 2022-03-20T15:00:14.953 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2022-03-20T15:00:14.978 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-03-20T15:00:14.984 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi006.front.sepia.ceph.com,smithi045.front.sepia.ceph.com,smithi123.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2022-03-20T15:03:37.824 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi006.front.sepia.ceph.com'), Remote(name='ubuntu@smithi045.front.sepia.ceph.com'), Remote(name='ubuntu@smithi123.front.sepia.ceph.com')] 2022-03-20T15:03:37.827 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2022-03-20T15:03:37.828 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-20T15:03:37.880 DEBUG:teuthology.orchestra.run.smithi006:> true 2022-03-20T15:03:38.008 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2022-03-20T15:03:38.009 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-20T15:03:38.059 DEBUG:teuthology.orchestra.run.smithi045:> true 2022-03-20T15:03:38.184 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2022-03-20T15:03:38.185 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-20T15:03:38.226 DEBUG:teuthology.orchestra.run.smithi123:> true 2022-03-20T15:03:38.344 INFO:teuthology.run_tasks:Running task clock... 2022-03-20T15:03:38.355 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-03-20T15:03:38.356 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-20T15:03:38.357 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-20T15:03:38.360 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-20T15:03:38.360 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-20T15:03:38.363 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-20T15:03:38.363 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-20T15:03:38.392 INFO:teuthology.orchestra.run.smithi006.stdout:20 Mar 15:03:38 ntpd[9397]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-20T15:03:38.393 INFO:teuthology.orchestra.run.smithi006.stdout:20 Mar 15:03:38 ntpd[9397]: Command line: ntpd -gq 2022-03-20T15:03:38.393 INFO:teuthology.orchestra.run.smithi045.stdout:20 Mar 15:03:38 ntpd[9397]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-20T15:03:38.394 INFO:teuthology.orchestra.run.smithi045.stdout:20 Mar 15:03:38 ntpd[9397]: Command line: ntpd -gq 2022-03-20T15:03:38.394 INFO:teuthology.orchestra.run.smithi006.stdout:20 Mar 15:03:38 ntpd[9397]: proto: precision = 0.103 usec (-23) 2022-03-20T15:03:38.395 INFO:teuthology.orchestra.run.smithi006.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-20T15:03:38.395 INFO:teuthology.orchestra.run.smithi006.stdout:20 Mar 15:03:38 ntpd[9397]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-20T15:03:38.396 INFO:teuthology.orchestra.run.smithi006.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-20T15:03:38.397 INFO:teuthology.orchestra.run.smithi006.stdout:20 Mar 15:03:38 ntpd[9397]: restrict ::: KOD does nothing without LIMITED. 2022-03-20T15:03:38.398 INFO:teuthology.orchestra.run.smithi006.stdout:20 Mar 15:03:38 ntpd[9397]: Listen and drop on 0 v6wildcard [::]:123 2022-03-20T15:03:38.398 INFO:teuthology.orchestra.run.smithi006.stdout:20 Mar 15:03:38 ntpd[9397]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-20T15:03:38.398 INFO:teuthology.orchestra.run.smithi006.stdout:20 Mar 15:03:38 ntpd[9397]: Listen normally on 2 lo 127.0.0.1:123 2022-03-20T15:03:38.398 INFO:teuthology.orchestra.run.smithi006.stdout:20 Mar 15:03:38 ntpd[9397]: Listen normally on 3 ens1f0 172.21.15.6:123 2022-03-20T15:03:38.399 INFO:teuthology.orchestra.run.smithi006.stdout:20 Mar 15:03:38 ntpd[9397]: Listen normally on 4 lo [::1]:123 2022-03-20T15:03:38.399 INFO:teuthology.orchestra.run.smithi006.stdout:20 Mar 15:03:38 ntpd[9397]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:15e4%4]:123 2022-03-20T15:03:38.399 INFO:teuthology.orchestra.run.smithi006.stdout:20 Mar 15:03:38 ntpd[9397]: Listening on routing socket on fd #22 for interface updates 2022-03-20T15:03:38.400 INFO:teuthology.orchestra.run.smithi045.stdout:20 Mar 15:03:38 ntpd[9397]: proto: precision = 0.080 usec (-23) 2022-03-20T15:03:38.401 INFO:teuthology.orchestra.run.smithi123.stdout:20 Mar 15:03:38 ntpd[9534]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-20T15:03:38.401 INFO:teuthology.orchestra.run.smithi123.stdout:20 Mar 15:03:38 ntpd[9534]: Command line: ntpd -gq 2022-03-20T15:03:38.401 INFO:teuthology.orchestra.run.smithi123.stdout:20 Mar 15:03:38 ntpd[9534]: proto: precision = 0.093 usec (-23) 2022-03-20T15:03:38.402 INFO:teuthology.orchestra.run.smithi045.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-20T15:03:38.402 INFO:teuthology.orchestra.run.smithi123.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-20T15:03:38.403 INFO:teuthology.orchestra.run.smithi045.stdout:20 Mar 15:03:38 ntpd[9397]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-20T15:03:38.403 INFO:teuthology.orchestra.run.smithi123.stdout:20 Mar 15:03:38 ntpd[9534]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-20T15:03:38.404 INFO:teuthology.orchestra.run.smithi045.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-20T15:03:38.404 INFO:teuthology.orchestra.run.smithi123.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-20T15:03:38.406 INFO:teuthology.orchestra.run.smithi045.stdout:20 Mar 15:03:38 ntpd[9397]: restrict ::: KOD does nothing without LIMITED. 2022-03-20T15:03:38.406 INFO:teuthology.orchestra.run.smithi045.stdout:20 Mar 15:03:38 ntpd[9397]: Listen and drop on 0 v6wildcard [::]:123 2022-03-20T15:03:38.407 INFO:teuthology.orchestra.run.smithi045.stdout:20 Mar 15:03:38 ntpd[9397]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-20T15:03:38.407 INFO:teuthology.orchestra.run.smithi045.stdout:20 Mar 15:03:38 ntpd[9397]: Listen normally on 2 lo 127.0.0.1:123 2022-03-20T15:03:38.407 INFO:teuthology.orchestra.run.smithi045.stdout:20 Mar 15:03:38 ntpd[9397]: Listen normally on 3 ens1f0 172.21.15.45:123 2022-03-20T15:03:38.407 INFO:teuthology.orchestra.run.smithi045.stdout:20 Mar 15:03:38 ntpd[9397]: Listen normally on 4 lo [::1]:123 2022-03-20T15:03:38.408 INFO:teuthology.orchestra.run.smithi045.stdout:20 Mar 15:03:38 ntpd[9397]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:12be%4]:123 2022-03-20T15:03:38.408 INFO:teuthology.orchestra.run.smithi045.stdout:20 Mar 15:03:38 ntpd[9397]: Listening on routing socket on fd #22 for interface updates 2022-03-20T15:03:38.408 INFO:teuthology.orchestra.run.smithi123.stdout:20 Mar 15:03:38 ntpd[9534]: restrict ::: KOD does nothing without LIMITED. 2022-03-20T15:03:38.409 INFO:teuthology.orchestra.run.smithi123.stdout:20 Mar 15:03:38 ntpd[9534]: Listen and drop on 0 v6wildcard [::]:123 2022-03-20T15:03:38.409 INFO:teuthology.orchestra.run.smithi123.stdout:20 Mar 15:03:38 ntpd[9534]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-20T15:03:38.409 INFO:teuthology.orchestra.run.smithi123.stdout:20 Mar 15:03:38 ntpd[9534]: Listen normally on 2 lo 127.0.0.1:123 2022-03-20T15:03:38.409 INFO:teuthology.orchestra.run.smithi123.stdout:20 Mar 15:03:38 ntpd[9534]: Listen normally on 3 enp3s0f1 172.21.15.123:123 2022-03-20T15:03:38.410 INFO:teuthology.orchestra.run.smithi123.stdout:20 Mar 15:03:38 ntpd[9534]: Listen normally on 4 lo [::1]:123 2022-03-20T15:03:38.410 INFO:teuthology.orchestra.run.smithi123.stdout:20 Mar 15:03:38 ntpd[9534]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d3f1%5]:123 2022-03-20T15:03:38.410 INFO:teuthology.orchestra.run.smithi123.stdout:20 Mar 15:03:38 ntpd[9534]: Listening on routing socket on fd #22 for interface updates 2022-03-20T15:03:39.394 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:39 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:39.394 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:39 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:39.394 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:39 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:39.395 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:39 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:39.395 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:39 ntpd[9397]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T15:03:39.395 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:39 ntpd[9397]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T15:03:39.396 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:39 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:39.396 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:39 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:39.397 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:39 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:39.397 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:39 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:39.397 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:39 ntpd[9397]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T15:03:39.397 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:39 ntpd[9397]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T15:03:39.398 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:39 ntpd[9534]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:39.399 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:39 ntpd[9534]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:39.399 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:39 ntpd[9534]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:39.400 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:39 ntpd[9534]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:39.400 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:39 ntpd[9534]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T15:03:39.400 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:39 ntpd[9534]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T15:03:40.393 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:40 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:40.394 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:40 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:40.394 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:40 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:40.394 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:40 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:40.394 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:40 ntpd[9397]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T15:03:40.395 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:40 ntpd[9397]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T15:03:40.396 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:40 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:40.396 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:40 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:40.396 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:40 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:40.397 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:40 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:40.397 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:40 ntpd[9397]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T15:03:40.397 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:40 ntpd[9397]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T15:03:40.400 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:40 ntpd[9534]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:40.400 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:40 ntpd[9534]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:40.401 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:40 ntpd[9534]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:40.401 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:40 ntpd[9534]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:40.401 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:40 ntpd[9534]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:40.401 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:40 ntpd[9534]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:40.402 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:40 ntpd[9534]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:40.402 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:40 ntpd[9534]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:40.402 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:40 ntpd[9534]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T15:03:40.402 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:40 ntpd[9534]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T15:03:40.402 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:40 ntpd[9534]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:40.403 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:40 ntpd[9534]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:40.403 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:40 ntpd[9534]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T15:03:40.403 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:40 ntpd[9534]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T15:03:40.403 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:40 ntpd[9534]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:40.404 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:40 ntpd[9534]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:40.404 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:40 ntpd[9534]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T15:03:40.404 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:40 ntpd[9534]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T15:03:41.393 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:41 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:41.394 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:41 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:41.394 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:41 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:41.394 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:41 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:41.394 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:41 ntpd[9397]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T15:03:41.395 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:41 ntpd[9397]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T15:03:41.396 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:41 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:41.396 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:41 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:41.396 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:41 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:41.396 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:41 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:41.397 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:41 ntpd[9397]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T15:03:41.397 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:41 ntpd[9397]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T15:03:41.399 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:41 ntpd[9534]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:41.399 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:41 ntpd[9534]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:41.400 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:41 ntpd[9534]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:41.400 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:41 ntpd[9534]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:41.400 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:41 ntpd[9534]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T15:03:41.400 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:41 ntpd[9534]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T15:03:42.394 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:42 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:42.394 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:42 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:42.395 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:42 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:42.395 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:42 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:42.395 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:42 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:42.395 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:42 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:42.396 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:42 ntpd[9397]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T15:03:42.396 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:42 ntpd[9397]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T15:03:42.396 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:42 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:42.396 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:42 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:42.396 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:42 ntpd[9397]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T15:03:42.397 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:42 ntpd[9397]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T15:03:42.399 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:42 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:42.399 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:42 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:42.400 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:42 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:42.400 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:42 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:42.400 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:42 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:42.400 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:42 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:42.400 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:42 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:42.401 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:42 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:42.401 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:42 ntpd[9397]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T15:03:42.401 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:42 ntpd[9397]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T15:03:42.401 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:42 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:42.402 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:42 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:42.402 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:42 ntpd[9397]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T15:03:42.402 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:42 ntpd[9397]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T15:03:42.402 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:42 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:42.403 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:42 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:42.403 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:42 ntpd[9397]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T15:03:42.403 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:42 ntpd[9397]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T15:03:42.405 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:42 ntpd[9534]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:42.406 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:42 ntpd[9534]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:42.406 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:42 ntpd[9534]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:42.406 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:42 ntpd[9534]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:42.408 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:42 ntpd[9534]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:42.408 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:42 ntpd[9534]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:42.408 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:42 ntpd[9534]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:42.409 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:42 ntpd[9534]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:42.410 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:42 ntpd[9534]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T15:03:42.410 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:42 ntpd[9534]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T15:03:42.410 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:42 ntpd[9534]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:42.410 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:42 ntpd[9534]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:42.411 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:42 ntpd[9534]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T15:03:42.411 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:42 ntpd[9534]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T15:03:42.411 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:42 ntpd[9534]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:42.411 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:42 ntpd[9534]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:42.411 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:42 ntpd[9534]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T15:03:42.412 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:42 ntpd[9534]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T15:03:43.394 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:43 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:43.394 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:43 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:43.395 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:43 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:43.395 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:43 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:43.395 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:43 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:43.395 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:43 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:43.396 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:43 ntpd[9397]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T15:03:43.396 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:43 ntpd[9397]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T15:03:43.396 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:43 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:43.396 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:43 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:43.396 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:43 ntpd[9397]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T15:03:43.397 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:43 ntpd[9397]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T15:03:43.398 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:43 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:43.398 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:43 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:43.398 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:43 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:43.399 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:43 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:43.399 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:43 ntpd[9397]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T15:03:43.399 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:43 ntpd[9397]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T15:03:43.400 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:43 ntpd[9534]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:43.400 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:43 ntpd[9534]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:43.401 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:43 ntpd[9534]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:43.401 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:43 ntpd[9534]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:43.401 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:43 ntpd[9534]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T15:03:43.401 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:43 ntpd[9534]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T15:03:44.394 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:44 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:44.394 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:44 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:44.394 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:44 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:44.395 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:44 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:44.395 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:44 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:44.395 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:44 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:44.395 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:44 ntpd[9397]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T15:03:44.396 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:44 ntpd[9397]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T15:03:44.396 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:44 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:44.396 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:44 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:44.396 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:44 ntpd[9397]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T15:03:44.397 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:44 ntpd[9397]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T15:03:44.399 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:44 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:44.399 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:44 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:44.399 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:44 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:44.400 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:44 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:44.400 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:44 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:44.400 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:44 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:44.400 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:44 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:44.401 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:44 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:44.401 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:44 ntpd[9397]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T15:03:44.401 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:44 ntpd[9397]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T15:03:44.401 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:44 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:44.401 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:44 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:44.402 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:44 ntpd[9397]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T15:03:44.402 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:44 ntpd[9397]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T15:03:44.402 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:44 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:44.403 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:44 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:44.403 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:44 ntpd[9397]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T15:03:44.403 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:44 ntpd[9397]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T15:03:44.405 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:44 ntpd[9534]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:44.406 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:44 ntpd[9534]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:44.406 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:44 ntpd[9534]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:44.406 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:44 ntpd[9534]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:44.407 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:44 ntpd[9534]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:44.407 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:44 ntpd[9534]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:44.407 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:44 ntpd[9534]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:44.407 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:44 ntpd[9534]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:44.407 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:44 ntpd[9534]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T15:03:44.408 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:44 ntpd[9534]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T15:03:44.408 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:44 ntpd[9534]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:44.408 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:44 ntpd[9534]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:44.408 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:44 ntpd[9534]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T15:03:44.409 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:44 ntpd[9534]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T15:03:44.409 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:44 ntpd[9534]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:44.409 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:44 ntpd[9534]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:44.409 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:44 ntpd[9534]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T15:03:44.409 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:44 ntpd[9534]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T15:03:45.394 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:45 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:45.394 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:45 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:45.394 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:45 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:45.395 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:45 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:45.395 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:45 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:45.395 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:45 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:45.395 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:45 ntpd[9397]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T15:03:45.396 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:45 ntpd[9397]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T15:03:45.396 INFO:teuthology.orchestra.run.smithi006.stdout:20 Mar 15:03:45 ntpd[9397]: ntpd: time slew -0.000522 s 2022-03-20T15:03:45.396 INFO:teuthology.orchestra.run.smithi006.stdout:ntpd: time slew -0.000522s 2022-03-20T15:03:45.397 INFO:teuthology.orchestra.run.smithi006.stderr:20 Mar 15:03:45 ntpd[9397]: can't open /var/log/ntpstats/loopstats.20220320: Permission denied 2022-03-20T15:03:45.398 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:45 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:45.399 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:45 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:45.399 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:45 ntpd[9397]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:45.399 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:45 ntpd[9397]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:45.399 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:45 ntpd[9397]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T15:03:45.400 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:45 ntpd[9397]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T15:03:45.400 INFO:teuthology.orchestra.run.smithi045.stdout:20 Mar 15:03:45 ntpd[9397]: ntpd: time slew -0.000791 s 2022-03-20T15:03:45.400 INFO:teuthology.orchestra.run.smithi045.stdout:ntpd: time slew -0.000791s 2022-03-20T15:03:45.401 INFO:teuthology.orchestra.run.smithi045.stderr:20 Mar 15:03:45 ntpd[9397]: can't open /var/log/ntpstats/loopstats.20220320: Permission denied 2022-03-20T15:03:45.402 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:45 ntpd[9534]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:45.403 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:45 ntpd[9534]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:45.403 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:45 ntpd[9534]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T15:03:45.403 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:45 ntpd[9534]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T15:03:45.404 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:45 ntpd[9534]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T15:03:45.404 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:45 ntpd[9534]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T15:03:45.404 INFO:teuthology.orchestra.run.smithi123.stdout:20 Mar 15:03:45 ntpd[9534]: ntpd: time slew -0.000089 s 2022-03-20T15:03:45.405 INFO:teuthology.orchestra.run.smithi123.stdout:ntpd: time slew -0.000089s 2022-03-20T15:03:45.405 INFO:teuthology.orchestra.run.smithi123.stderr:20 Mar 15:03:45 ntpd[9534]: can't open /var/log/ntpstats/loopstats.20220320: Permission denied 2022-03-20T15:03:45.453 INFO:teuthology.orchestra.run.smithi045.stdout: remote refid st t when poll reach delay offset jitter 2022-03-20T15:03:45.454 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================== 2022-03-20T15:03:45.454 INFO:teuthology.orchestra.run.smithi045.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-20T15:03:45.454 INFO:teuthology.orchestra.run.smithi045.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-20T15:03:45.455 INFO:teuthology.orchestra.run.smithi045.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-20T15:03:45.455 INFO:teuthology.orchestra.run.smithi045.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-20T15:03:45.456 INFO:teuthology.orchestra.run.smithi006.stdout: remote refid st t when poll reach delay offset jitter 2022-03-20T15:03:45.456 INFO:teuthology.orchestra.run.smithi006.stdout:============================================================================== 2022-03-20T15:03:45.456 INFO:teuthology.orchestra.run.smithi006.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-20T15:03:45.457 INFO:teuthology.orchestra.run.smithi006.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-20T15:03:45.457 INFO:teuthology.orchestra.run.smithi006.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-20T15:03:45.457 INFO:teuthology.orchestra.run.smithi006.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-20T15:03:45.459 INFO:teuthology.orchestra.run.smithi123.stdout: remote refid st t when poll reach delay offset jitter 2022-03-20T15:03:45.460 INFO:teuthology.orchestra.run.smithi123.stdout:============================================================================== 2022-03-20T15:03:45.460 INFO:teuthology.orchestra.run.smithi123.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-20T15:03:45.460 INFO:teuthology.orchestra.run.smithi123.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-20T15:03:45.460 INFO:teuthology.orchestra.run.smithi123.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-20T15:03:45.461 INFO:teuthology.orchestra.run.smithi123.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-20T15:03:45.462 INFO:teuthology.run_tasks:Running task kubeadm... 2022-03-20T15:03:45.478 INFO:tasks.kubeadm:Kubeadm start 2022-03-20T15:03:45.479 INFO:tasks.kubeadm:Config: {'pod_network': 'calico', 'version': '1.21'} 2022-03-20T15:03:45.479 DEBUG:teuthology.orchestra.run.smithi006:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-20T15:03:45.503 DEBUG:teuthology.orchestra.run.smithi045:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-20T15:03:45.505 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-20T15:03:45.528 INFO:teuthology.orchestra.run.smithi006.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-20T15:03:45.529 INFO:teuthology.orchestra.run.smithi045.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-20T15:03:45.530 INFO:teuthology.orchestra.run.smithi123.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-20T15:03:45.538 INFO:teuthology.orchestra.run.smithi045.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-20T15:03:45.538 INFO:teuthology.orchestra.run.smithi123.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-20T15:03:45.539 INFO:teuthology.orchestra.run.smithi006.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-20T15:03:45.544 INFO:teuthology.orchestra.run.smithi045.stdout:net.ipv4.ip_forward = 1 2022-03-20T15:03:45.544 INFO:teuthology.orchestra.run.smithi123.stdout:net.ipv4.ip_forward = 1 2022-03-20T15:03:45.547 INFO:teuthology.orchestra.run.smithi006.stdout:net.ipv4.ip_forward = 1 2022-03-20T15:03:45.653 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2022-03-20T15:03:45.653 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-20T15:03:45.669 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2022-03-20T15:03:45.669 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/docker/daemon.json 2022-03-20T15:03:45.680 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2022-03-20T15:03:45.681 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-20T15:03:45.710 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2022-03-20T15:03:45.712 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/docker/daemon.json 2022-03-20T15:03:45.727 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2022-03-20T15:03:45.727 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-20T15:03:45.745 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2022-03-20T15:03:45.746 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/docker/daemon.json 2022-03-20T15:03:45.757 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl restart docker || true 2022-03-20T15:03:45.760 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl restart docker || true 2022-03-20T15:03:45.772 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl restart docker || true 2022-03-20T15:03:47.031 INFO:tasks.kubeadm:Identifying pod subnet 2022-03-20T15:03:47.038 INFO:tasks.kubeadm:172.21.15.6 is in 172.21.0.0/20 at pos 3845 2022-03-20T15:03:47.051 INFO:tasks.kubeadm:Pod subnet: 10.248.40.0/21 2022-03-20T15:03:47.051 INFO:tasks.kubeadm:Installing kube{adm,ctl,let} 2022-03-20T15:03:47.052 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-20T15:03:47.055 DEBUG:teuthology.orchestra.run.smithi045:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-20T15:03:47.057 DEBUG:teuthology.orchestra.run.smithi123:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-20T15:03:47.081 INFO:teuthology.orchestra.run.smithi006.stderr: 2022-03-20T15:03:47.081 INFO:teuthology.orchestra.run.smithi006.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-20T15:03:47.082 INFO:teuthology.orchestra.run.smithi006.stderr: 2022-03-20T15:03:47.085 INFO:teuthology.orchestra.run.smithi045.stderr: 2022-03-20T15:03:47.085 INFO:teuthology.orchestra.run.smithi045.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-20T15:03:47.085 INFO:teuthology.orchestra.run.smithi045.stderr: 2022-03-20T15:03:47.089 INFO:teuthology.orchestra.run.smithi123.stderr: 2022-03-20T15:03:47.089 INFO:teuthology.orchestra.run.smithi123.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-20T15:03:47.089 INFO:teuthology.orchestra.run.smithi123.stderr: 2022-03-20T15:03:47.198 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-20T15:03:47.202 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-20T15:03:47.220 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-20T15:03:47.228 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-20T15:03:47.248 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-20T15:03:47.308 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-20T15:03:47.328 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-20T15:03:47.330 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-20T15:03:47.408 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-20T15:03:47.409 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-20T15:03:47.489 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-20T15:03:47.495 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-20T15:03:48.682 INFO:teuthology.orchestra.run.smithi123.stdout:Reading package lists... 2022-03-20T15:03:48.725 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2022-03-20T15:03:48.740 INFO:teuthology.orchestra.run.smithi045.stdout:Reading package lists... 2022-03-20T15:03:48.850 INFO:teuthology.orchestra.run.smithi123.stdout:Building dependency tree... 2022-03-20T15:03:48.851 INFO:teuthology.orchestra.run.smithi123.stdout:Reading state information... 2022-03-20T15:03:48.902 INFO:teuthology.orchestra.run.smithi006.stdout:Building dependency tree... 2022-03-20T15:03:48.902 INFO:teuthology.orchestra.run.smithi006.stdout:Reading state information... 2022-03-20T15:03:48.917 INFO:teuthology.orchestra.run.smithi045.stdout:Building dependency tree... 2022-03-20T15:03:48.918 INFO:teuthology.orchestra.run.smithi045.stdout:Reading state information... 2022-03-20T15:03:48.942 INFO:teuthology.orchestra.run.smithi123.stdout:98 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-20T15:03:48.957 INFO:teuthology.orchestra.run.smithi123.stderr: 2022-03-20T15:03:48.957 INFO:teuthology.orchestra.run.smithi123.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-20T15:03:48.958 INFO:teuthology.orchestra.run.smithi123.stderr: 2022-03-20T15:03:48.992 INFO:teuthology.orchestra.run.smithi006.stdout:98 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-20T15:03:49.007 INFO:teuthology.orchestra.run.smithi006.stderr: 2022-03-20T15:03:49.007 INFO:teuthology.orchestra.run.smithi006.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-20T15:03:49.007 INFO:teuthology.orchestra.run.smithi006.stderr: 2022-03-20T15:03:49.011 INFO:teuthology.orchestra.run.smithi045.stdout:98 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-20T15:03:49.014 INFO:teuthology.orchestra.run.smithi123.stdout:Reading package lists... 2022-03-20T15:03:49.028 INFO:teuthology.orchestra.run.smithi045.stderr: 2022-03-20T15:03:49.028 INFO:teuthology.orchestra.run.smithi045.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-20T15:03:49.029 INFO:teuthology.orchestra.run.smithi045.stderr: 2022-03-20T15:03:49.061 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2022-03-20T15:03:49.082 INFO:teuthology.orchestra.run.smithi045.stdout:Reading package lists... 2022-03-20T15:03:49.189 INFO:teuthology.orchestra.run.smithi123.stdout:Building dependency tree... 2022-03-20T15:03:49.189 INFO:teuthology.orchestra.run.smithi123.stdout:Reading state information... 2022-03-20T15:03:49.241 INFO:teuthology.orchestra.run.smithi006.stdout:Building dependency tree... 2022-03-20T15:03:49.242 INFO:teuthology.orchestra.run.smithi006.stdout:Reading state information... 2022-03-20T15:03:49.260 INFO:teuthology.orchestra.run.smithi045.stdout:Building dependency tree... 2022-03-20T15:03:49.261 INFO:teuthology.orchestra.run.smithi045.stdout:Reading state information... 2022-03-20T15:03:49.361 INFO:teuthology.orchestra.run.smithi123.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-20T15:03:49.361 INFO:teuthology.orchestra.run.smithi123.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-20T15:03:49.361 INFO:teuthology.orchestra.run.smithi123.stdout:The following NEW packages will be installed: 2022-03-20T15:03:49.362 INFO:teuthology.orchestra.run.smithi123.stdout: apt-transport-https 2022-03-20T15:03:49.421 INFO:teuthology.orchestra.run.smithi123.stdout:0 upgraded, 1 newly installed, 0 to remove and 98 not upgraded. 2022-03-20T15:03:49.421 INFO:teuthology.orchestra.run.smithi123.stdout:Need to get 4,680 B of archives. 2022-03-20T15:03:49.421 INFO:teuthology.orchestra.run.smithi123.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-20T15:03:49.422 INFO:teuthology.orchestra.run.smithi123.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-20T15:03:49.431 INFO:teuthology.orchestra.run.smithi006.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-20T15:03:49.432 INFO:teuthology.orchestra.run.smithi006.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-20T15:03:49.432 INFO:teuthology.orchestra.run.smithi006.stdout:The following NEW packages will be installed: 2022-03-20T15:03:49.433 INFO:teuthology.orchestra.run.smithi006.stdout: apt-transport-https 2022-03-20T15:03:49.451 INFO:teuthology.orchestra.run.smithi045.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-20T15:03:49.451 INFO:teuthology.orchestra.run.smithi045.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-20T15:03:49.451 INFO:teuthology.orchestra.run.smithi045.stdout:The following NEW packages will be installed: 2022-03-20T15:03:49.452 INFO:teuthology.orchestra.run.smithi045.stdout: apt-transport-https 2022-03-20T15:03:49.494 INFO:teuthology.orchestra.run.smithi006.stdout:0 upgraded, 1 newly installed, 0 to remove and 98 not upgraded. 2022-03-20T15:03:49.494 INFO:teuthology.orchestra.run.smithi006.stdout:Need to get 4,680 B of archives. 2022-03-20T15:03:49.495 INFO:teuthology.orchestra.run.smithi006.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-20T15:03:49.495 INFO:teuthology.orchestra.run.smithi006.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-20T15:03:49.513 INFO:teuthology.orchestra.run.smithi045.stdout:0 upgraded, 1 newly installed, 0 to remove and 98 not upgraded. 2022-03-20T15:03:49.513 INFO:teuthology.orchestra.run.smithi045.stdout:Need to get 4,680 B of archives. 2022-03-20T15:03:49.513 INFO:teuthology.orchestra.run.smithi045.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-20T15:03:49.513 INFO:teuthology.orchestra.run.smithi045.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-20T15:03:49.640 INFO:teuthology.orchestra.run.smithi123.stderr:debconf: unable to initialize frontend: Dialog 2022-03-20T15:03:49.641 INFO:teuthology.orchestra.run.smithi123.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-20T15:03:49.641 INFO:teuthology.orchestra.run.smithi123.stderr:debconf: falling back to frontend: Readline 2022-03-20T15:03:49.662 INFO:teuthology.orchestra.run.smithi123.stderr:debconf: unable to initialize frontend: Readline 2022-03-20T15:03:49.662 INFO:teuthology.orchestra.run.smithi123.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-20T15:03:49.662 INFO:teuthology.orchestra.run.smithi123.stderr:debconf: falling back to frontend: Teletype 2022-03-20T15:03:49.668 INFO:teuthology.orchestra.run.smithi123.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-20T15:03:49.719 INFO:teuthology.orchestra.run.smithi123.stdout:Fetched 4,680 B in 0s (89.4 kB/s) 2022-03-20T15:03:49.720 INFO:teuthology.orchestra.run.smithi006.stderr:debconf: unable to initialize frontend: Dialog 2022-03-20T15:03:49.720 INFO:teuthology.orchestra.run.smithi006.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-20T15:03:49.720 INFO:teuthology.orchestra.run.smithi006.stderr:debconf: falling back to frontend: Readline 2022-03-20T15:03:49.746 INFO:teuthology.orchestra.run.smithi006.stderr:debconf: unable to initialize frontend: Readline 2022-03-20T15:03:49.746 INFO:teuthology.orchestra.run.smithi006.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-20T15:03:49.746 INFO:teuthology.orchestra.run.smithi006.stderr:debconf: falling back to frontend: Teletype 2022-03-20T15:03:49.752 INFO:teuthology.orchestra.run.smithi045.stderr:debconf: unable to initialize frontend: Dialog 2022-03-20T15:03:49.752 INFO:teuthology.orchestra.run.smithi045.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-20T15:03:49.752 INFO:teuthology.orchestra.run.smithi045.stderr:debconf: falling back to frontend: Readline 2022-03-20T15:03:49.752 INFO:teuthology.orchestra.run.smithi006.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-20T15:03:49.775 INFO:teuthology.orchestra.run.smithi045.stderr:debconf: unable to initialize frontend: Readline 2022-03-20T15:03:49.775 INFO:teuthology.orchestra.run.smithi045.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-20T15:03:49.776 INFO:teuthology.orchestra.run.smithi045.stderr:debconf: falling back to frontend: Teletype 2022-03-20T15:03:49.781 INFO:teuthology.orchestra.run.smithi045.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-20T15:03:49.807 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package apt-transport-https. 2022-03-20T15:03:49.807 INFO:teuthology.orchestra.run.smithi006.stdout:Fetched 4,680 B in 0s (87.2 kB/s) 2022-03-20T15:03:49.834 INFO:teuthology.orchestra.run.smithi045.stdout:Fetched 4,680 B in 0s (88.8 kB/s) 2022-03-20T15:03:49.851 INFO:teuthology.orchestra.run.smithi123.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-20T15:03:49.854 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-20T15:03:49.895 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package apt-transport-https. 2022-03-20T15:03:49.913 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-20T15:03:49.919 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package apt-transport-https. 2022-03-20T15:03:49.940 INFO:teuthology.orchestra.run.smithi006.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-20T15:03:49.944 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-20T15:03:49.964 INFO:teuthology.orchestra.run.smithi045.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-20T15:03:49.967 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-20T15:03:49.987 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-20T15:03:50.009 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-20T15:03:50.140 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-20T15:03:50.249 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-20T15:03:50.298 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-20T15:03:50.691 INFO:teuthology.orchestra.run.smithi123.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-20T15:03:50.707 INFO:teuthology.orchestra.run.smithi123.stderr: 2022-03-20T15:03:50.707 INFO:teuthology.orchestra.run.smithi123.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-20T15:03:50.707 INFO:teuthology.orchestra.run.smithi123.stderr: 2022-03-20T15:03:50.786 INFO:teuthology.orchestra.run.smithi006.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-20T15:03:50.793 INFO:teuthology.orchestra.run.smithi045.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-20T15:03:50.800 INFO:teuthology.orchestra.run.smithi006.stderr: 2022-03-20T15:03:50.800 INFO:teuthology.orchestra.run.smithi006.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-20T15:03:50.800 INFO:teuthology.orchestra.run.smithi006.stderr: 2022-03-20T15:03:50.807 INFO:teuthology.orchestra.run.smithi045.stderr: 2022-03-20T15:03:50.808 INFO:teuthology.orchestra.run.smithi045.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-20T15:03:50.808 INFO:teuthology.orchestra.run.smithi045.stderr: 2022-03-20T15:03:50.838 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-20T15:03:50.838 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-20T15:03:50.846 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-20T15:03:50.866 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-20T15:03:50.928 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-20T15:03:50.937 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-20T15:03:50.942 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-20T15:03:50.943 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-20T15:03:50.951 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-20T15:03:50.957 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-20T15:03:50.971 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-20T15:03:51.038 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:5 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-20T15:03:51.327 INFO:teuthology.orchestra.run.smithi123.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-20T15:03:51.468 INFO:teuthology.orchestra.run.smithi123.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [54.7 kB] 2022-03-20T15:03:51.538 INFO:teuthology.orchestra.run.smithi006.stdout:Get:4 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-20T15:03:51.606 INFO:teuthology.orchestra.run.smithi123.stdout:Fetched 64.1 kB in 1s (85.0 kB/s) 2022-03-20T15:03:51.606 INFO:teuthology.orchestra.run.smithi045.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-20T15:03:51.653 INFO:teuthology.orchestra.run.smithi006.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [54.7 kB] 2022-03-20T15:03:51.704 INFO:teuthology.orchestra.run.smithi045.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [54.7 kB] 2022-03-20T15:03:51.789 INFO:teuthology.orchestra.run.smithi006.stdout:Fetched 64.1 kB in 1s (76.3 kB/s) 2022-03-20T15:03:51.847 INFO:teuthology.orchestra.run.smithi045.stdout:Fetched 64.1 kB in 1s (72.4 kB/s) 2022-03-20T15:03:52.456 INFO:teuthology.orchestra.run.smithi123.stdout:Reading package lists... 2022-03-20T15:03:52.627 INFO:teuthology.orchestra.run.smithi123.stdout:Building dependency tree... 2022-03-20T15:03:52.627 INFO:teuthology.orchestra.run.smithi123.stdout:Reading state information... 2022-03-20T15:03:52.671 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2022-03-20T15:03:52.716 INFO:teuthology.orchestra.run.smithi123.stdout:98 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-20T15:03:52.727 INFO:teuthology.orchestra.run.smithi045.stdout:Reading package lists... 2022-03-20T15:03:52.733 INFO:teuthology.orchestra.run.smithi123.stderr: 2022-03-20T15:03:52.733 INFO:teuthology.orchestra.run.smithi123.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-20T15:03:52.733 INFO:teuthology.orchestra.run.smithi123.stderr: 2022-03-20T15:03:52.787 INFO:teuthology.orchestra.run.smithi123.stdout:Reading package lists... 2022-03-20T15:03:52.849 INFO:teuthology.orchestra.run.smithi006.stdout:Building dependency tree... 2022-03-20T15:03:52.850 INFO:teuthology.orchestra.run.smithi006.stdout:Reading state information... 2022-03-20T15:03:52.909 INFO:teuthology.orchestra.run.smithi045.stdout:Building dependency tree... 2022-03-20T15:03:52.909 INFO:teuthology.orchestra.run.smithi045.stdout:Reading state information... 2022-03-20T15:03:52.945 INFO:teuthology.orchestra.run.smithi006.stdout:98 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-20T15:03:52.957 INFO:teuthology.orchestra.run.smithi006.stderr: 2022-03-20T15:03:52.958 INFO:teuthology.orchestra.run.smithi006.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-20T15:03:52.958 INFO:teuthology.orchestra.run.smithi006.stderr: 2022-03-20T15:03:52.962 INFO:teuthology.orchestra.run.smithi123.stdout:Building dependency tree... 2022-03-20T15:03:52.963 INFO:teuthology.orchestra.run.smithi123.stdout:Reading state information... 2022-03-20T15:03:53.012 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2022-03-20T15:03:53.013 INFO:teuthology.orchestra.run.smithi045.stdout:98 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-20T15:03:53.028 INFO:teuthology.orchestra.run.smithi045.stderr: 2022-03-20T15:03:53.028 INFO:teuthology.orchestra.run.smithi045.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-20T15:03:53.028 INFO:teuthology.orchestra.run.smithi045.stderr: 2022-03-20T15:03:53.087 INFO:teuthology.orchestra.run.smithi045.stdout:Reading package lists... 2022-03-20T15:03:53.104 INFO:teuthology.orchestra.run.smithi123.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-20T15:03:53.104 INFO:teuthology.orchestra.run.smithi123.stdout:bridge-utils set to manually installed. 2022-03-20T15:03:53.104 INFO:teuthology.orchestra.run.smithi123.stdout:The following additional packages will be installed: 2022-03-20T15:03:53.105 INFO:teuthology.orchestra.run.smithi123.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-20T15:03:53.105 INFO:teuthology.orchestra.run.smithi123.stdout:Suggested packages: 2022-03-20T15:03:53.106 INFO:teuthology.orchestra.run.smithi123.stdout: nftables 2022-03-20T15:03:53.154 INFO:teuthology.orchestra.run.smithi123.stdout:The following NEW packages will be installed: 2022-03-20T15:03:53.155 INFO:teuthology.orchestra.run.smithi123.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-20T15:03:53.192 INFO:teuthology.orchestra.run.smithi006.stdout:Building dependency tree... 2022-03-20T15:03:53.193 INFO:teuthology.orchestra.run.smithi006.stdout:Reading state information... 2022-03-20T15:03:53.223 INFO:teuthology.orchestra.run.smithi123.stdout:0 upgraded, 8 newly installed, 0 to remove and 98 not upgraded. 2022-03-20T15:03:53.223 INFO:teuthology.orchestra.run.smithi123.stdout:Need to get 77.7 MB of archives. 2022-03-20T15:03:53.223 INFO:teuthology.orchestra.run.smithi123.stdout:After this operation, 335 MB of additional disk space will be used. 2022-03-20T15:03:53.224 INFO:teuthology.orchestra.run.smithi123.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-20T15:03:53.263 INFO:teuthology.orchestra.run.smithi123.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-20T15:03:53.268 INFO:teuthology.orchestra.run.smithi045.stdout:Building dependency tree... 2022-03-20T15:03:53.269 INFO:teuthology.orchestra.run.smithi045.stdout:Reading state information... 2022-03-20T15:03:53.291 INFO:teuthology.orchestra.run.smithi123.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-20T15:03:53.342 INFO:teuthology.orchestra.run.smithi006.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-20T15:03:53.343 INFO:teuthology.orchestra.run.smithi006.stdout:bridge-utils set to manually installed. 2022-03-20T15:03:53.343 INFO:teuthology.orchestra.run.smithi006.stdout:The following additional packages will be installed: 2022-03-20T15:03:53.344 INFO:teuthology.orchestra.run.smithi006.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-20T15:03:53.344 INFO:teuthology.orchestra.run.smithi006.stdout:Suggested packages: 2022-03-20T15:03:53.344 INFO:teuthology.orchestra.run.smithi006.stdout: nftables 2022-03-20T15:03:53.392 INFO:teuthology.orchestra.run.smithi006.stdout:The following NEW packages will be installed: 2022-03-20T15:03:53.393 INFO:teuthology.orchestra.run.smithi006.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-20T15:03:53.422 INFO:teuthology.orchestra.run.smithi045.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-20T15:03:53.422 INFO:teuthology.orchestra.run.smithi045.stdout:bridge-utils set to manually installed. 2022-03-20T15:03:53.423 INFO:teuthology.orchestra.run.smithi045.stdout:The following additional packages will be installed: 2022-03-20T15:03:53.423 INFO:teuthology.orchestra.run.smithi045.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-20T15:03:53.424 INFO:teuthology.orchestra.run.smithi045.stdout:Suggested packages: 2022-03-20T15:03:53.424 INFO:teuthology.orchestra.run.smithi045.stdout: nftables 2022-03-20T15:03:53.431 INFO:teuthology.orchestra.run.smithi123.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.23.0-00 [15.3 MB] 2022-03-20T15:03:53.462 INFO:teuthology.orchestra.run.smithi006.stdout:0 upgraded, 8 newly installed, 0 to remove and 98 not upgraded. 2022-03-20T15:03:53.463 INFO:teuthology.orchestra.run.smithi006.stdout:Need to get 77.7 MB of archives. 2022-03-20T15:03:53.463 INFO:teuthology.orchestra.run.smithi006.stdout:After this operation, 335 MB of additional disk space will be used. 2022-03-20T15:03:53.463 INFO:teuthology.orchestra.run.smithi006.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-20T15:03:53.474 INFO:teuthology.orchestra.run.smithi045.stdout:The following NEW packages will be installed: 2022-03-20T15:03:53.476 INFO:teuthology.orchestra.run.smithi045.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-20T15:03:53.503 INFO:teuthology.orchestra.run.smithi006.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-20T15:03:53.530 INFO:teuthology.orchestra.run.smithi006.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-20T15:03:53.546 INFO:teuthology.orchestra.run.smithi045.stdout:0 upgraded, 8 newly installed, 0 to remove and 98 not upgraded. 2022-03-20T15:03:53.546 INFO:teuthology.orchestra.run.smithi045.stdout:Need to get 77.7 MB of archives. 2022-03-20T15:03:53.547 INFO:teuthology.orchestra.run.smithi045.stdout:After this operation, 335 MB of additional disk space will be used. 2022-03-20T15:03:53.547 INFO:teuthology.orchestra.run.smithi045.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-20T15:03:53.591 INFO:teuthology.orchestra.run.smithi045.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-20T15:03:53.617 INFO:teuthology.orchestra.run.smithi045.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-20T15:03:53.637 INFO:teuthology.orchestra.run.smithi006.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.23.0-00 [15.3 MB] 2022-03-20T15:03:53.752 INFO:teuthology.orchestra.run.smithi045.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.23.0-00 [15.3 MB] 2022-03-20T15:03:54.118 INFO:teuthology.orchestra.run.smithi123.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-20T15:03:54.206 INFO:teuthology.orchestra.run.smithi006.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-20T15:03:55.094 INFO:teuthology.orchestra.run.smithi006.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.5-00 [19.5 MB] 2022-03-20T15:03:55.105 INFO:teuthology.orchestra.run.smithi045.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-20T15:03:55.119 INFO:teuthology.orchestra.run.smithi123.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.5-00 [19.5 MB] 2022-03-20T15:03:55.841 INFO:teuthology.orchestra.run.smithi123.stdout:Get:7 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.5-00 [8,931 kB] 2022-03-20T15:03:55.898 INFO:teuthology.orchestra.run.smithi006.stdout:Get:7 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.5-00 [8,931 kB] 2022-03-20T15:03:56.165 INFO:teuthology.orchestra.run.smithi123.stdout:Get:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.5-00 [8,582 kB] 2022-03-20T15:03:56.225 INFO:teuthology.orchestra.run.smithi006.stdout:Get:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.5-00 [8,582 kB] 2022-03-20T15:03:56.594 INFO:teuthology.orchestra.run.smithi123.stderr:debconf: unable to initialize frontend: Dialog 2022-03-20T15:03:56.595 INFO:teuthology.orchestra.run.smithi123.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-20T15:03:56.595 INFO:teuthology.orchestra.run.smithi123.stderr:debconf: falling back to frontend: Readline 2022-03-20T15:03:56.602 INFO:teuthology.orchestra.run.smithi123.stderr:debconf: unable to initialize frontend: Readline 2022-03-20T15:03:56.603 INFO:teuthology.orchestra.run.smithi123.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-20T15:03:56.603 INFO:teuthology.orchestra.run.smithi123.stderr:debconf: falling back to frontend: Teletype 2022-03-20T15:03:56.609 INFO:teuthology.orchestra.run.smithi123.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-20T15:03:56.656 INFO:teuthology.orchestra.run.smithi006.stderr:debconf: unable to initialize frontend: Dialog 2022-03-20T15:03:56.656 INFO:teuthology.orchestra.run.smithi006.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-20T15:03:56.657 INFO:teuthology.orchestra.run.smithi006.stderr:debconf: falling back to frontend: Readline 2022-03-20T15:03:56.657 INFO:teuthology.orchestra.run.smithi123.stdout:Fetched 77.7 MB in 3s (24.1 MB/s) 2022-03-20T15:03:56.665 INFO:teuthology.orchestra.run.smithi006.stderr:debconf: unable to initialize frontend: Readline 2022-03-20T15:03:56.665 INFO:teuthology.orchestra.run.smithi006.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-20T15:03:56.665 INFO:teuthology.orchestra.run.smithi006.stderr:debconf: falling back to frontend: Teletype 2022-03-20T15:03:56.671 INFO:teuthology.orchestra.run.smithi006.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-20T15:03:56.722 INFO:teuthology.orchestra.run.smithi006.stdout:Fetched 77.7 MB in 3s (25.7 MB/s) 2022-03-20T15:03:56.755 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package conntrack. 2022-03-20T15:03:56.797 INFO:teuthology.orchestra.run.smithi123.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-20T15:03:56.800 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-20T15:03:56.811 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package conntrack. 2022-03-20T15:03:56.845 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-20T15:03:56.850 INFO:teuthology.orchestra.run.smithi006.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-20T15:03:56.853 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-20T15:03:56.894 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-20T15:03:57.065 INFO:teuthology.orchestra.run.smithi045.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.5-00 [19.5 MB] 2022-03-20T15:03:57.150 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package cri-tools. 2022-03-20T15:03:57.166 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../1-cri-tools_1.23.0-00_amd64.deb ... 2022-03-20T15:03:57.172 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package cri-tools. 2022-03-20T15:03:57.184 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../1-cri-tools_1.23.0-00_amd64.deb ... 2022-03-20T15:03:57.209 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking cri-tools (1.23.0-00) ... 2022-03-20T15:03:57.222 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking cri-tools (1.23.0-00) ... 2022-03-20T15:03:58.106 INFO:teuthology.orchestra.run.smithi045.stdout:Get:7 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.5-00 [8,931 kB] 2022-03-20T15:03:58.558 INFO:teuthology.orchestra.run.smithi045.stdout:Get:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.5-00 [8,582 kB] 2022-03-20T15:03:58.570 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ebtables. 2022-03-20T15:03:58.582 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-20T15:03:58.635 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-20T15:03:58.684 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ebtables. 2022-03-20T15:03:58.697 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-20T15:03:58.744 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-20T15:03:58.865 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-20T15:03:58.879 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-20T15:03:58.923 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-20T15:03:58.990 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-20T15:03:59.003 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-20T15:03:59.048 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-20T15:03:59.298 INFO:teuthology.orchestra.run.smithi045.stderr:debconf: unable to initialize frontend: Dialog 2022-03-20T15:03:59.298 INFO:teuthology.orchestra.run.smithi045.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-20T15:03:59.299 INFO:teuthology.orchestra.run.smithi045.stderr:debconf: falling back to frontend: Readline 2022-03-20T15:03:59.307 INFO:teuthology.orchestra.run.smithi045.stderr:debconf: unable to initialize frontend: Readline 2022-03-20T15:03:59.307 INFO:teuthology.orchestra.run.smithi045.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-20T15:03:59.308 INFO:teuthology.orchestra.run.smithi045.stderr:debconf: falling back to frontend: Teletype 2022-03-20T15:03:59.313 INFO:teuthology.orchestra.run.smithi045.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-20T15:03:59.366 INFO:teuthology.orchestra.run.smithi045.stdout:Fetched 77.7 MB in 6s (13.9 MB/s) 2022-03-20T15:03:59.466 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package conntrack. 2022-03-20T15:03:59.511 INFO:teuthology.orchestra.run.smithi045.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-20T15:03:59.514 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-20T15:03:59.564 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-20T15:03:59.883 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package cri-tools. 2022-03-20T15:03:59.897 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../1-cri-tools_1.23.0-00_amd64.deb ... 2022-03-20T15:03:59.941 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking cri-tools (1.23.0-00) ... 2022-03-20T15:04:00.384 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package socat. 2022-03-20T15:04:00.398 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-20T15:04:00.477 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-20T15:04:00.525 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package socat. 2022-03-20T15:04:00.539 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-20T15:04:00.583 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-20T15:04:00.850 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package kubelet. 2022-03-20T15:04:00.865 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../5-kubelet_1.23.5-00_amd64.deb ... 2022-03-20T15:04:00.908 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking kubelet (1.23.5-00) ... 2022-03-20T15:04:00.963 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package kubelet. 2022-03-20T15:04:00.976 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../5-kubelet_1.23.5-00_amd64.deb ... 2022-03-20T15:04:01.012 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking kubelet (1.23.5-00) ... 2022-03-20T15:04:01.447 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ebtables. 2022-03-20T15:04:01.461 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-20T15:04:01.513 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-20T15:04:01.750 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-20T15:04:01.763 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-20T15:04:01.808 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-20T15:04:03.261 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package kubectl. 2022-03-20T15:04:03.275 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../6-kubectl_1.23.5-00_amd64.deb ... 2022-03-20T15:04:03.286 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package socat. 2022-03-20T15:04:03.300 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-20T15:04:03.344 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-20T15:04:03.352 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking kubectl (1.23.5-00) ... 2022-03-20T15:04:03.398 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package kubectl. 2022-03-20T15:04:03.412 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../6-kubectl_1.23.5-00_amd64.deb ... 2022-03-20T15:04:03.456 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking kubectl (1.23.5-00) ... 2022-03-20T15:04:03.731 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package kubelet. 2022-03-20T15:04:03.745 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../5-kubelet_1.23.5-00_amd64.deb ... 2022-03-20T15:04:03.789 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking kubelet (1.23.5-00) ... 2022-03-20T15:04:04.555 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package kubeadm. 2022-03-20T15:04:04.570 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../7-kubeadm_1.23.5-00_amd64.deb ... 2022-03-20T15:04:04.613 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking kubeadm (1.23.5-00) ... 2022-03-20T15:04:04.620 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package kubeadm. 2022-03-20T15:04:04.633 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../7-kubeadm_1.23.5-00_amd64.deb ... 2022-03-20T15:04:04.669 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking kubeadm (1.23.5-00) ... 2022-03-20T15:04:05.731 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-20T15:04:05.800 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-20T15:04:05.852 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up kubectl (1.23.5-00) ... 2022-03-20T15:04:05.932 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up kubectl (1.23.5-00) ... 2022-03-20T15:04:05.979 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-20T15:04:06.050 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-20T15:04:06.333 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package kubectl. 2022-03-20T15:04:06.349 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../6-kubectl_1.23.5-00_amd64.deb ... 2022-03-20T15:04:06.408 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking kubectl (1.23.5-00) ... 2022-03-20T15:04:06.803 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-20T15:04:06.853 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-20T15:04:06.955 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up cri-tools (1.23.0-00) ... 2022-03-20T15:04:06.988 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up cri-tools (1.23.0-00) ... 2022-03-20T15:04:07.083 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-20T15:04:07.116 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-20T15:04:07.210 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up kubelet (1.23.5-00) ... 2022-03-20T15:04:07.226 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up kubelet (1.23.5-00) ... 2022-03-20T15:04:07.428 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-20T15:04:07.464 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-20T15:04:07.595 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package kubeadm. 2022-03-20T15:04:07.609 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../7-kubeadm_1.23.5-00_amd64.deb ... 2022-03-20T15:04:07.645 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking kubeadm (1.23.5-00) ... 2022-03-20T15:04:07.955 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up kubeadm (1.23.5-00) ... 2022-03-20T15:04:07.966 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up kubeadm (1.23.5-00) ... 2022-03-20T15:04:08.360 INFO:teuthology.orchestra.run.smithi006.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-20T15:04:08.372 INFO:teuthology.orchestra.run.smithi123.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-20T15:04:08.908 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-20T15:04:09.033 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up kubectl (1.23.5-00) ... 2022-03-20T15:04:09.175 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-20T15:04:09.962 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-20T15:04:10.081 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up cri-tools (1.23.0-00) ... 2022-03-20T15:04:10.208 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-20T15:04:10.360 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up kubelet (1.23.5-00) ... 2022-03-20T15:04:10.575 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-20T15:04:11.097 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up kubeadm (1.23.5-00) ... 2022-03-20T15:04:11.618 INFO:teuthology.orchestra.run.smithi045.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-20T15:04:12.519 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-20T15:04:12.522 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-20T15:04:12.525 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-20T15:04:22.815 INFO:teuthology.orchestra.run.smithi006.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.5 2022-03-20T15:04:22.895 INFO:teuthology.orchestra.run.smithi123.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.5 2022-03-20T15:04:25.833 INFO:teuthology.orchestra.run.smithi006.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.5 2022-03-20T15:04:26.014 INFO:teuthology.orchestra.run.smithi123.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.5 2022-03-20T15:04:26.560 INFO:teuthology.orchestra.run.smithi045.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.5 2022-03-20T15:04:28.165 INFO:teuthology.orchestra.run.smithi006.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.5 2022-03-20T15:04:28.194 INFO:teuthology.orchestra.run.smithi123.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.5 2022-03-20T15:04:29.304 INFO:teuthology.orchestra.run.smithi045.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.5 2022-03-20T15:04:31.573 INFO:teuthology.orchestra.run.smithi006.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.5 2022-03-20T15:04:31.760 INFO:teuthology.orchestra.run.smithi045.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.5 2022-03-20T15:04:31.902 INFO:teuthology.orchestra.run.smithi123.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.5 2022-03-20T15:04:32.972 INFO:teuthology.orchestra.run.smithi006.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-20T15:04:33.216 INFO:teuthology.orchestra.run.smithi123.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-20T15:04:35.676 INFO:teuthology.orchestra.run.smithi045.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.5 2022-03-20T15:04:36.998 INFO:teuthology.orchestra.run.smithi045.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-20T15:04:39.071 INFO:teuthology.orchestra.run.smithi006.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-20T15:04:39.209 INFO:teuthology.orchestra.run.smithi123.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-20T15:04:41.287 INFO:teuthology.orchestra.run.smithi006.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-20T15:04:41.808 INFO:teuthology.orchestra.run.smithi123.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-20T15:04:42.342 INFO:teuthology.orchestra.run.smithi045.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-20T15:04:44.440 INFO:teuthology.orchestra.run.smithi045.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-20T15:04:44.444 INFO:tasks.kubeadm:Token: abcdef.fp7pk6o84z6o4e9z 2022-03-20T15:04:44.445 INFO:tasks.kubeadm:Remotes: {Remote(name='ubuntu@smithi006.front.sepia.ceph.com'): '172.21.15.6', Remote(name='ubuntu@smithi045.front.sepia.ceph.com'): '172.21.15.45', Remote(name='ubuntu@smithi123.front.sepia.ceph.com'): '172.21.15.123'} 2022-03-20T15:04:44.445 DEBUG:teuthology.orchestra.run.smithi006:> sudo kubeadm init --node-name smithi006 --token abcdef.fp7pk6o84z6o4e9z --pod-network-cidr 10.248.40.0/21 2022-03-20T15:04:44.656 INFO:teuthology.orchestra.run.smithi006.stdout:[init] Using Kubernetes version: v1.23.5 2022-03-20T15:04:44.657 INFO:teuthology.orchestra.run.smithi006.stdout:[preflight] Running pre-flight checks 2022-03-20T15:04:45.072 INFO:teuthology.orchestra.run.smithi006.stdout:[preflight] Pulling images required for setting up a Kubernetes cluster 2022-03-20T15:04:45.073 INFO:teuthology.orchestra.run.smithi006.stdout:[preflight] This might take a minute or two, depending on the speed of your internet connection 2022-03-20T15:04:45.073 INFO:teuthology.orchestra.run.smithi006.stdout:[preflight] You can also perform this action in beforehand using 'kubeadm config images pull' 2022-03-20T15:04:45.379 INFO:teuthology.orchestra.run.smithi006.stdout:[certs] Using certificateDir folder "/etc/kubernetes/pki" 2022-03-20T15:04:45.980 INFO:teuthology.orchestra.run.smithi006.stdout:[certs] Generating "ca" certificate and key 2022-03-20T15:04:46.185 INFO:teuthology.orchestra.run.smithi006.stdout:[certs] Generating "apiserver" certificate and key 2022-03-20T15:04:46.186 INFO:teuthology.orchestra.run.smithi006.stdout:[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local smithi006] and IPs [10.96.0.1 172.21.15.6] 2022-03-20T15:04:46.285 INFO:teuthology.orchestra.run.smithi006.stdout:[certs] Generating "apiserver-kubelet-client" certificate and key 2022-03-20T15:04:46.391 INFO:teuthology.orchestra.run.smithi006.stdout:[certs] Generating "front-proxy-ca" certificate and key 2022-03-20T15:04:46.549 INFO:teuthology.orchestra.run.smithi006.stdout:[certs] Generating "front-proxy-client" certificate and key 2022-03-20T15:04:46.780 INFO:teuthology.orchestra.run.smithi006.stdout:[certs] Generating "etcd/ca" certificate and key 2022-03-20T15:04:46.946 INFO:teuthology.orchestra.run.smithi006.stdout:[certs] Generating "etcd/server" certificate and key 2022-03-20T15:04:46.947 INFO:teuthology.orchestra.run.smithi006.stdout:[certs] etcd/server serving cert is signed for DNS names [localhost smithi006] and IPs [172.21.15.6 127.0.0.1 ::1] 2022-03-20T15:04:47.188 INFO:teuthology.orchestra.run.smithi006.stdout:[certs] Generating "etcd/peer" certificate and key 2022-03-20T15:04:47.188 INFO:teuthology.orchestra.run.smithi006.stdout:[certs] etcd/peer serving cert is signed for DNS names [localhost smithi006] and IPs [172.21.15.6 127.0.0.1 ::1] 2022-03-20T15:04:47.255 INFO:teuthology.orchestra.run.smithi006.stdout:[certs] Generating "etcd/healthcheck-client" certificate and key 2022-03-20T15:04:47.576 INFO:teuthology.orchestra.run.smithi006.stdout:[certs] Generating "apiserver-etcd-client" certificate and key 2022-03-20T15:04:47.731 INFO:teuthology.orchestra.run.smithi006.stdout:[certs] Generating "sa" key and public key 2022-03-20T15:04:47.731 INFO:teuthology.orchestra.run.smithi006.stdout:[kubeconfig] Using kubeconfig folder "/etc/kubernetes" 2022-03-20T15:04:47.872 INFO:teuthology.orchestra.run.smithi006.stdout:[kubeconfig] Writing "admin.conf" kubeconfig file 2022-03-20T15:04:48.035 INFO:teuthology.orchestra.run.smithi006.stdout:[kubeconfig] Writing "kubelet.conf" kubeconfig file 2022-03-20T15:04:48.198 INFO:teuthology.orchestra.run.smithi006.stdout:[kubeconfig] Writing "controller-manager.conf" kubeconfig file 2022-03-20T15:04:48.426 INFO:teuthology.orchestra.run.smithi006.stdout:[kubeconfig] Writing "scheduler.conf" kubeconfig file 2022-03-20T15:04:48.442 INFO:teuthology.orchestra.run.smithi006.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-20T15:04:48.444 INFO:teuthology.orchestra.run.smithi006.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-20T15:04:48.444 INFO:teuthology.orchestra.run.smithi006.stdout:[kubelet-start] Starting the kubelet 2022-03-20T15:04:48.642 INFO:teuthology.orchestra.run.smithi006.stdout:[control-plane] Using manifest folder "/etc/kubernetes/manifests" 2022-03-20T15:04:48.642 INFO:teuthology.orchestra.run.smithi006.stdout:[control-plane] Creating static Pod manifest for "kube-apiserver" 2022-03-20T15:04:48.646 INFO:teuthology.orchestra.run.smithi006.stdout:[control-plane] Creating static Pod manifest for "kube-controller-manager" 2022-03-20T15:04:48.647 INFO:teuthology.orchestra.run.smithi006.stdout:[control-plane] Creating static Pod manifest for "kube-scheduler" 2022-03-20T15:04:48.647 INFO:teuthology.orchestra.run.smithi006.stdout:[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" 2022-03-20T15:04:48.649 INFO:teuthology.orchestra.run.smithi006.stdout:[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s 2022-03-20T15:05:05.652 INFO:teuthology.orchestra.run.smithi006.stdout:[apiclient] All control plane components are healthy after 17.002940 seconds 2022-03-20T15:05:05.653 INFO:teuthology.orchestra.run.smithi006.stdout:[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace 2022-03-20T15:05:05.672 INFO:teuthology.orchestra.run.smithi006.stdout:[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster 2022-03-20T15:05:05.672 INFO:teuthology.orchestra.run.smithi006.stdout:NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently. 2022-03-20T15:05:06.257 INFO:teuthology.orchestra.run.smithi006.stdout:[upload-certs] Skipping phase. Please see --upload-certs 2022-03-20T15:05:06.257 INFO:teuthology.orchestra.run.smithi006.stdout:[mark-control-plane] Marking the node smithi006 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] 2022-03-20T15:05:06.258 INFO:teuthology.orchestra.run.smithi006.stdout:[mark-control-plane] Marking the node smithi006 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] 2022-03-20T15:05:06.789 INFO:teuthology.orchestra.run.smithi006.stdout:[bootstrap-token] Using token: abcdef.fp7pk6o84z6o4e9z 2022-03-20T15:05:06.789 INFO:teuthology.orchestra.run.smithi006.stdout:[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles 2022-03-20T15:05:06.816 INFO:teuthology.orchestra.run.smithi006.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes 2022-03-20T15:05:06.830 INFO:teuthology.orchestra.run.smithi006.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials 2022-03-20T15:05:06.838 INFO:teuthology.orchestra.run.smithi006.stdout:[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token 2022-03-20T15:05:06.847 INFO:teuthology.orchestra.run.smithi006.stdout:[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster 2022-03-20T15:05:06.872 INFO:teuthology.orchestra.run.smithi006.stdout:[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace 2022-03-20T15:05:06.938 INFO:teuthology.orchestra.run.smithi006.stdout:[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key 2022-03-20T15:05:07.382 INFO:teuthology.orchestra.run.smithi006.stdout:[addons] Applied essential addon: CoreDNS 2022-03-20T15:05:07.541 INFO:teuthology.orchestra.run.smithi006.stdout:[addons] Applied essential addon: kube-proxy 2022-03-20T15:05:07.542 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:05:07.542 INFO:teuthology.orchestra.run.smithi006.stdout:Your Kubernetes control-plane has initialized successfully! 2022-03-20T15:05:07.543 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:05:07.543 INFO:teuthology.orchestra.run.smithi006.stdout:To start using your cluster, you need to run the following as a regular user: 2022-03-20T15:05:07.543 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:05:07.543 INFO:teuthology.orchestra.run.smithi006.stdout: mkdir -p $HOME/.kube 2022-03-20T15:05:07.544 INFO:teuthology.orchestra.run.smithi006.stdout: sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 2022-03-20T15:05:07.544 INFO:teuthology.orchestra.run.smithi006.stdout: sudo chown $(id -u):$(id -g) $HOME/.kube/config 2022-03-20T15:05:07.544 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:05:07.545 INFO:teuthology.orchestra.run.smithi006.stdout:Alternatively, if you are the root user, you can run: 2022-03-20T15:05:07.545 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:05:07.545 INFO:teuthology.orchestra.run.smithi006.stdout: export KUBECONFIG=/etc/kubernetes/admin.conf 2022-03-20T15:05:07.545 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:05:07.546 INFO:teuthology.orchestra.run.smithi006.stdout:You should now deploy a pod network to the cluster. 2022-03-20T15:05:07.546 INFO:teuthology.orchestra.run.smithi006.stdout:Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: 2022-03-20T15:05:07.546 INFO:teuthology.orchestra.run.smithi006.stdout: https://kubernetes.io/docs/concepts/cluster-administration/addons/ 2022-03-20T15:05:07.546 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:05:07.547 INFO:teuthology.orchestra.run.smithi006.stdout:Then you can join any number of worker nodes by running the following on each as root: 2022-03-20T15:05:07.547 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:05:07.547 INFO:teuthology.orchestra.run.smithi006.stdout:kubeadm join 172.21.15.6:6443 --token abcdef.fp7pk6o84z6o4e9z \ 2022-03-20T15:05:07.547 INFO:teuthology.orchestra.run.smithi006.stdout: --discovery-token-ca-cert-hash sha256:b266e362d01b41328eae13e3b9fbb302c12f29c4e05f16f5a37ecf56056f5f5d 2022-03-20T15:05:07.549 DEBUG:teuthology.orchestra.run.smithi045:> sudo kubeadm join 172.21.15.6:6443 --node-name smithi045 --token abcdef.fp7pk6o84z6o4e9z --discovery-token-unsafe-skip-ca-verification 2022-03-20T15:05:07.552 DEBUG:teuthology.orchestra.run.smithi123:> sudo kubeadm join 172.21.15.6:6443 --node-name smithi123 --token abcdef.fp7pk6o84z6o4e9z --discovery-token-unsafe-skip-ca-verification 2022-03-20T15:05:07.587 INFO:teuthology.orchestra.run.smithi045.stdout:[preflight] Running pre-flight checks 2022-03-20T15:05:07.593 INFO:teuthology.orchestra.run.smithi123.stdout:[preflight] Running pre-flight checks 2022-03-20T15:05:26.381 INFO:teuthology.orchestra.run.smithi045.stdout:[preflight] Reading configuration from the cluster... 2022-03-20T15:05:26.381 INFO:teuthology.orchestra.run.smithi045.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-03-20T15:05:26.382 INFO:teuthology.orchestra.run.smithi123.stdout:[preflight] Reading configuration from the cluster... 2022-03-20T15:05:26.382 INFO:teuthology.orchestra.run.smithi123.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-03-20T15:05:26.732 INFO:teuthology.orchestra.run.smithi123.stderr:W0320 15:05:26.734212 11169 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf 2022-03-20T15:05:26.733 INFO:teuthology.orchestra.run.smithi045.stderr:W0320 15:05:26.740400 11002 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf 2022-03-20T15:05:26.750 INFO:teuthology.orchestra.run.smithi123.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-20T15:05:26.750 INFO:teuthology.orchestra.run.smithi123.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-20T15:05:26.750 INFO:teuthology.orchestra.run.smithi123.stdout:[kubelet-start] Starting the kubelet 2022-03-20T15:05:26.751 INFO:teuthology.orchestra.run.smithi045.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-20T15:05:26.752 INFO:teuthology.orchestra.run.smithi045.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-20T15:05:26.752 INFO:teuthology.orchestra.run.smithi045.stdout:[kubelet-start] Starting the kubelet 2022-03-20T15:05:26.964 INFO:teuthology.orchestra.run.smithi123.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-03-20T15:05:26.966 INFO:teuthology.orchestra.run.smithi045.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-03-20T15:05:32.502 INFO:teuthology.orchestra.run.smithi123.stdout: 2022-03-20T15:05:32.502 INFO:teuthology.orchestra.run.smithi123.stdout:This node has joined the cluster: 2022-03-20T15:05:32.502 INFO:teuthology.orchestra.run.smithi123.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-03-20T15:05:32.503 INFO:teuthology.orchestra.run.smithi123.stdout:* The Kubelet was informed of the new secure connection details. 2022-03-20T15:05:32.503 INFO:teuthology.orchestra.run.smithi123.stdout: 2022-03-20T15:05:32.503 INFO:teuthology.orchestra.run.smithi123.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-03-20T15:05:32.503 INFO:teuthology.orchestra.run.smithi123.stdout: 2022-03-20T15:05:32.504 INFO:teuthology.orchestra.run.smithi045.stdout: 2022-03-20T15:05:32.504 INFO:teuthology.orchestra.run.smithi045.stdout:This node has joined the cluster: 2022-03-20T15:05:32.504 INFO:teuthology.orchestra.run.smithi045.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-03-20T15:05:32.505 INFO:teuthology.orchestra.run.smithi045.stdout:* The Kubelet was informed of the new secure connection details. 2022-03-20T15:05:32.505 INFO:teuthology.orchestra.run.smithi045.stdout: 2022-03-20T15:05:32.505 INFO:teuthology.orchestra.run.smithi045.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-03-20T15:05:32.505 INFO:teuthology.orchestra.run.smithi045.stdout: 2022-03-20T15:05:32.508 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2022-03-20T15:05:32.509 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/etc/kubernetes/admin.conf of=/dev/stdout 2022-03-20T15:05:32.521 INFO:tasks.kubeadm:Setting up kubectl 2022-03-20T15:05:32.522 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-20T15:05:32.537 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-20T15:05:32.555 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-20T15:05:32.571 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2022-03-20T15:05:32.572 DEBUG:teuthology.orchestra.run.smithi006:> dd of=.kube/config 2022-03-20T15:05:32.579 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2022-03-20T15:05:32.580 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/root/.kube/config 2022-03-20T15:05:32.634 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2022-03-20T15:05:32.634 DEBUG:teuthology.orchestra.run.smithi045:> dd of=.kube/config 2022-03-20T15:05:32.642 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2022-03-20T15:05:32.642 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/root/.kube/config 2022-03-20T15:05:32.699 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2022-03-20T15:05:32.700 DEBUG:teuthology.orchestra.run.smithi123:> dd of=.kube/config 2022-03-20T15:05:32.707 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2022-03-20T15:05:32.708 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/root/.kube/config 2022-03-20T15:05:32.765 DEBUG:teuthology.orchestra.run.smithi006:> kubectl apply -f https://docs.projectcalico.org/manifests/tigera-operator.yaml 2022-03-20T15:05:35.189 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created 2022-03-20T15:05:35.238 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created 2022-03-20T15:05:35.323 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created 2022-03-20T15:05:35.359 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created 2022-03-20T15:05:35.407 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created 2022-03-20T15:05:35.508 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created 2022-03-20T15:05:35.563 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created 2022-03-20T15:05:35.644 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created 2022-03-20T15:05:35.699 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created 2022-03-20T15:05:35.840 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created 2022-03-20T15:05:35.966 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created 2022-03-20T15:05:35.990 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created 2022-03-20T15:05:36.040 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created 2022-03-20T15:05:36.116 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created 2022-03-20T15:05:36.160 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created 2022-03-20T15:05:36.216 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created 2022-03-20T15:05:36.295 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created 2022-03-20T15:05:36.341 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/apiservers.operator.tigera.io created 2022-03-20T15:05:36.447 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/imagesets.operator.tigera.io created 2022-03-20T15:05:36.509 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/installations.operator.tigera.io created 2022-03-20T15:05:36.601 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/tigerastatuses.operator.tigera.io created 2022-03-20T15:05:36.666 INFO:teuthology.orchestra.run.smithi006.stdout:namespace/tigera-operator created 2022-03-20T15:05:36.677 INFO:teuthology.orchestra.run.smithi006.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-20T15:05:36.755 INFO:teuthology.orchestra.run.smithi006.stdout:podsecuritypolicy.policy/tigera-operator created 2022-03-20T15:05:36.762 INFO:teuthology.orchestra.run.smithi006.stdout:serviceaccount/tigera-operator created 2022-03-20T15:05:36.817 INFO:teuthology.orchestra.run.smithi006.stdout:clusterrole.rbac.authorization.k8s.io/tigera-operator created 2022-03-20T15:05:36.844 INFO:teuthology.orchestra.run.smithi006.stdout:clusterrolebinding.rbac.authorization.k8s.io/tigera-operator created 2022-03-20T15:05:36.892 INFO:teuthology.orchestra.run.smithi006.stdout:deployment.apps/tigera-operator created 2022-03-20T15:05:36.898 DEBUG:teuthology.orchestra.run.smithi006:> kubectl create -f - 2022-03-20T15:05:39.826 INFO:teuthology.orchestra.run.smithi006.stdout:installation.operator.tigera.io/default created 2022-03-20T15:05:39.834 INFO:tasks.kubeadm:Scanning for scratch devices 2022-03-20T15:05:39.835 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2022-03-20T15:05:39.835 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/scratch_devs of=/dev/stdout 2022-03-20T15:05:39.843 INFO:tasks.kubeadm:Scratch devices on smithi006: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-20T15:05:39.843 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-20T15:05:39.870 INFO:teuthology.orchestra.run.smithi006.stderr:10+0 records in 2022-03-20T15:05:39.870 INFO:teuthology.orchestra.run.smithi006.stderr:10+0 records out 2022-03-20T15:05:39.870 INFO:teuthology.orchestra.run.smithi006.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.015282 s, 686 MB/s 2022-03-20T15:05:39.871 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-20T15:05:39.896 INFO:teuthology.orchestra.run.smithi006.stderr:10+0 records in 2022-03-20T15:05:39.897 INFO:teuthology.orchestra.run.smithi006.stderr:10+0 records out 2022-03-20T15:05:39.897 INFO:teuthology.orchestra.run.smithi006.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0150283 s, 698 MB/s 2022-03-20T15:05:39.898 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-20T15:05:39.925 INFO:teuthology.orchestra.run.smithi006.stderr:10+0 records in 2022-03-20T15:05:39.926 INFO:teuthology.orchestra.run.smithi006.stderr:10+0 records out 2022-03-20T15:05:39.926 INFO:teuthology.orchestra.run.smithi006.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0158783 s, 660 MB/s 2022-03-20T15:05:39.927 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-20T15:05:39.956 INFO:teuthology.orchestra.run.smithi006.stderr:10+0 records in 2022-03-20T15:05:39.956 INFO:teuthology.orchestra.run.smithi006.stderr:10+0 records out 2022-03-20T15:05:39.957 INFO:teuthology.orchestra.run.smithi006.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0147527 s, 711 MB/s 2022-03-20T15:05:39.958 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2022-03-20T15:05:39.958 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/scratch_devs of=/dev/stdout 2022-03-20T15:05:39.967 INFO:tasks.kubeadm:Scratch devices on smithi045: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-20T15:05:39.967 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-20T15:05:40.044 INFO:teuthology.orchestra.run.smithi045.stderr:10+0 records in 2022-03-20T15:05:40.044 INFO:teuthology.orchestra.run.smithi045.stderr:10+0 records out 2022-03-20T15:05:40.045 INFO:teuthology.orchestra.run.smithi045.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0157336 s, 666 MB/s 2022-03-20T15:05:40.046 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-20T15:05:40.071 INFO:teuthology.orchestra.run.smithi045.stderr:10+0 records in 2022-03-20T15:05:40.071 INFO:teuthology.orchestra.run.smithi045.stderr:10+0 records out 2022-03-20T15:05:40.071 INFO:teuthology.orchestra.run.smithi045.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0124742 s, 841 MB/s 2022-03-20T15:05:40.072 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-20T15:05:40.100 INFO:teuthology.orchestra.run.smithi045.stderr:10+0 records in 2022-03-20T15:05:40.100 INFO:teuthology.orchestra.run.smithi045.stderr:10+0 records out 2022-03-20T15:05:40.100 INFO:teuthology.orchestra.run.smithi045.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0137458 s, 763 MB/s 2022-03-20T15:05:40.102 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-20T15:05:40.174 INFO:teuthology.orchestra.run.smithi045.stderr:10+0 records in 2022-03-20T15:05:40.174 INFO:teuthology.orchestra.run.smithi045.stderr:10+0 records out 2022-03-20T15:05:40.175 INFO:teuthology.orchestra.run.smithi045.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0138168 s, 759 MB/s 2022-03-20T15:05:40.176 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2022-03-20T15:05:40.176 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/scratch_devs of=/dev/stdout 2022-03-20T15:05:40.184 INFO:tasks.kubeadm:Scratch devices on smithi123: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-20T15:05:40.185 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-20T15:05:40.260 INFO:teuthology.orchestra.run.smithi123.stderr:10+0 records in 2022-03-20T15:05:40.260 INFO:teuthology.orchestra.run.smithi123.stderr:10+0 records out 2022-03-20T15:05:40.260 INFO:teuthology.orchestra.run.smithi123.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0183058 s, 573 MB/s 2022-03-20T15:05:40.262 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-20T15:05:40.287 INFO:teuthology.orchestra.run.smithi123.stderr:10+0 records in 2022-03-20T15:05:40.288 INFO:teuthology.orchestra.run.smithi123.stderr:10+0 records out 2022-03-20T15:05:40.288 INFO:teuthology.orchestra.run.smithi123.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.015851 s, 662 MB/s 2022-03-20T15:05:40.289 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-20T15:05:40.319 INFO:teuthology.orchestra.run.smithi123.stderr:10+0 records in 2022-03-20T15:05:40.319 INFO:teuthology.orchestra.run.smithi123.stderr:10+0 records out 2022-03-20T15:05:40.320 INFO:teuthology.orchestra.run.smithi123.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0165949 s, 632 MB/s 2022-03-20T15:05:40.321 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-20T15:05:40.348 INFO:teuthology.orchestra.run.smithi123.stderr:10+0 records in 2022-03-20T15:05:40.349 INFO:teuthology.orchestra.run.smithi123.stderr:10+0 records out 2022-03-20T15:05:40.349 INFO:teuthology.orchestra.run.smithi123.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0171458 s, 612 MB/s 2022-03-20T15:05:40.367 INFO:tasks.kubeadm:Creating PVs + StorageClass 2022-03-20T15:05:40.367 DEBUG:tasks.kubeadm:apiVersion: v1 kind: PersistentVolume metadata: name: smithi006-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi006 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi006-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi006 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi006-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi006 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi006-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi006 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi045-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi045 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi045-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi045 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi045-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi045 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi045-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi045 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi123-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi123 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi123-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi123 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi123-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi123 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi123-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi123 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: scratch provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer 2022-03-20T15:05:40.368 DEBUG:teuthology.orchestra.run.smithi006:> kubectl create -f - 2022-03-20T15:05:41.266 INFO:teuthology.orchestra.run.smithi006.stdout:persistentvolume/smithi006-lv-1 created 2022-03-20T15:05:41.277 INFO:teuthology.orchestra.run.smithi006.stdout:persistentvolume/smithi006-lv-2 created 2022-03-20T15:05:41.285 INFO:teuthology.orchestra.run.smithi006.stdout:persistentvolume/smithi006-lv-3 created 2022-03-20T15:05:41.307 INFO:teuthology.orchestra.run.smithi006.stdout:persistentvolume/smithi006-lv-4 created 2022-03-20T15:05:41.319 INFO:teuthology.orchestra.run.smithi006.stdout:persistentvolume/smithi045-lv-1 created 2022-03-20T15:05:41.327 INFO:teuthology.orchestra.run.smithi006.stdout:persistentvolume/smithi045-lv-2 created 2022-03-20T15:05:41.379 INFO:teuthology.orchestra.run.smithi006.stdout:persistentvolume/smithi045-lv-3 created 2022-03-20T15:05:41.382 INFO:teuthology.orchestra.run.smithi006.stdout:persistentvolume/smithi045-lv-4 created 2022-03-20T15:05:41.403 INFO:teuthology.orchestra.run.smithi006.stdout:persistentvolume/smithi123-lv-1 created 2022-03-20T15:05:41.423 INFO:teuthology.orchestra.run.smithi006.stdout:persistentvolume/smithi123-lv-2 created 2022-03-20T15:05:41.435 INFO:teuthology.orchestra.run.smithi006.stdout:persistentvolume/smithi123-lv-3 created 2022-03-20T15:05:41.457 INFO:teuthology.orchestra.run.smithi006.stdout:persistentvolume/smithi123-lv-4 created 2022-03-20T15:05:41.469 INFO:teuthology.orchestra.run.smithi006.stdout:storageclass.storage.k8s.io/scratch created 2022-03-20T15:05:41.473 DEBUG:teuthology.orchestra.run.smithi006:> kubectl taint node smithi006 node-role.kubernetes.io/master- || true 2022-03-20T15:05:41.544 INFO:teuthology.orchestra.run.smithi006.stdout:node/smithi006 untainted 2022-03-20T15:05:41.547 INFO:tasks.kubeadm:Kubeadm complete, yielding 2022-03-20T15:05:41.547 INFO:teuthology.run_tasks:Running task nvme_loop... 2022-03-20T15:05:41.562 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2022-03-20T15:05:41.562 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2022-03-20T15:05:41.563 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/scratch_devs of=/dev/stdout 2022-03-20T15:05:41.570 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-20T15:05:41.571 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-20T15:05:41.576 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-20T15:05:41.577 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-20T15:05:41.577 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 500 Links: 1 2022-03-20T15:05:41.577 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-20T15:05:41.578 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2022-03-20 15:05:39.870538223 +0000 2022-03-20T15:05:41.578 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2022-03-20 15:05:39.870538223 +0000 2022-03-20T15:05:41.578 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2022-03-20 15:05:39.870538223 +0000 2022-03-20T15:05:41.578 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: - 2022-03-20T15:05:41.585 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2022-03-20T15:05:41.585 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2022-03-20T15:05:41.586 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000596201 s, 859 kB/s 2022-03-20T15:05:41.590 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-20T15:05:41.642 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-20T15:05:41.642 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-20T15:05:41.642 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 505 Links: 1 2022-03-20T15:05:41.643 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-20T15:05:41.643 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2022-03-20 15:05:39.898538224 +0000 2022-03-20T15:05:41.643 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2022-03-20 15:05:39.898538224 +0000 2022-03-20T15:05:41.643 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2022-03-20 15:05:39.898538224 +0000 2022-03-20T15:05:41.644 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: - 2022-03-20T15:05:41.652 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2022-03-20T15:05:41.652 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2022-03-20T15:05:41.652 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000554001 s, 924 kB/s 2022-03-20T15:05:41.656 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-20T15:05:41.662 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-20T15:05:41.662 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-20T15:05:41.662 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 512 Links: 1 2022-03-20T15:05:41.663 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-20T15:05:41.663 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2022-03-20 15:05:39.926538224 +0000 2022-03-20T15:05:41.663 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2022-03-20 15:05:39.926538224 +0000 2022-03-20T15:05:41.663 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2022-03-20 15:05:39.926538224 +0000 2022-03-20T15:05:41.664 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: - 2022-03-20T15:05:41.672 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2022-03-20T15:05:41.672 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2022-03-20T15:05:41.673 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000570137 s, 898 kB/s 2022-03-20T15:05:41.677 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-20T15:05:41.684 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-20T15:05:41.684 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-20T15:05:41.684 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 519 Links: 1 2022-03-20T15:05:41.684 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-20T15:05:41.685 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2022-03-20 15:05:39.958538225 +0000 2022-03-20T15:05:41.685 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2022-03-20 15:05:39.958538225 +0000 2022-03-20T15:05:41.685 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2022-03-20 15:05:39.958538225 +0000 2022-03-20T15:05:41.685 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: - 2022-03-20T15:05:41.693 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2022-03-20T15:05:41.694 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2022-03-20T15:05:41.694 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000513113 s, 998 kB/s 2022-03-20T15:05:41.698 DEBUG:teuthology.orchestra.run.smithi006:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-20T15:05:41.794 INFO:teuthology.orchestra.run.smithi006.stdout:loop 2022-03-20T15:05:41.796 INFO:tasks.nvme_loop:Connecting nvme_loop smithi006:/dev/vg_nvme/lv_1... 2022-03-20T15:05:41.796 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-20T15:05:41.822 INFO:teuthology.orchestra.run.smithi006.stdout:1 2022-03-20T15:05:41.849 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/vg_nvme/lv_11 2022-03-20T15:05:41.872 INFO:tasks.nvme_loop:Connecting nvme_loop smithi006:/dev/vg_nvme/lv_2... 2022-03-20T15:05:41.873 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-20T15:05:41.943 INFO:teuthology.orchestra.run.smithi006.stdout:1 2022-03-20T15:05:41.964 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/vg_nvme/lv_21 2022-03-20T15:05:41.987 INFO:tasks.nvme_loop:Connecting nvme_loop smithi006:/dev/vg_nvme/lv_3... 2022-03-20T15:05:41.987 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-20T15:05:42.011 INFO:teuthology.orchestra.run.smithi006.stdout:1 2022-03-20T15:05:42.034 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/vg_nvme/lv_31 2022-03-20T15:05:42.056 INFO:tasks.nvme_loop:Connecting nvme_loop smithi006:/dev/vg_nvme/lv_4... 2022-03-20T15:05:42.057 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-20T15:05:42.078 INFO:teuthology.orchestra.run.smithi006.stdout:1 2022-03-20T15:05:42.108 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/vg_nvme/lv_41 2022-03-20T15:05:42.129 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2022-03-20T15:05:42.130 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/scratch_devs of=/dev/stdout 2022-03-20T15:05:42.177 DEBUG:teuthology.orchestra.run.smithi006:> sudo nvme list 2022-03-20T15:05:42.189 INFO:teuthology.orchestra.run.smithi006.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-20T15:05:42.189 INFO:teuthology.orchestra.run.smithi006.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-20T15:05:42.190 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/nvme0n1 CVFT5330000H400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-20T15:05:42.190 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/nvme1n1 17103b2322f87f36 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-20T15:05:42.190 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/nvme2n1 1f6c18f5a5be00cd Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-20T15:05:42.190 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/nvme3n1 09e248190860c6d8 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-20T15:05:42.191 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/nvme4n1 3892c9d405dc6252 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-20T15:05:42.192 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-20T15:05:42.192 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2022-03-20T15:05:42.192 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/scratch_devs 2022-03-20T15:05:42.208 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2022-03-20T15:05:42.208 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/scratch_devs of=/dev/stdout 2022-03-20T15:05:42.216 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-20T15:05:42.216 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-20T15:05:42.224 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-20T15:05:42.224 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-20T15:05:42.224 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 502 Links: 1 2022-03-20T15:05:42.225 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-20T15:05:42.225 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2022-03-20 15:05:40.049575899 +0000 2022-03-20T15:05:42.225 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2022-03-20 15:05:40.049575899 +0000 2022-03-20T15:05:42.225 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2022-03-20 15:05:40.049575899 +0000 2022-03-20T15:05:42.226 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: - 2022-03-20T15:05:42.233 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2022-03-20T15:05:42.233 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2022-03-20T15:05:42.234 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000296709 s, 1.7 MB/s 2022-03-20T15:05:42.238 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-20T15:05:42.292 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-20T15:05:42.292 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-20T15:05:42.293 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 508 Links: 1 2022-03-20T15:05:42.293 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-20T15:05:42.293 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2022-03-20 15:05:40.077575899 +0000 2022-03-20T15:05:42.293 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2022-03-20 15:05:40.077575899 +0000 2022-03-20T15:05:42.293 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2022-03-20 15:05:40.077575899 +0000 2022-03-20T15:05:42.294 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: - 2022-03-20T15:05:42.300 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2022-03-20T15:05:42.301 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2022-03-20T15:05:42.301 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000327411 s, 1.6 MB/s 2022-03-20T15:05:42.305 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-20T15:05:42.311 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-20T15:05:42.311 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-20T15:05:42.311 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 513 Links: 1 2022-03-20T15:05:42.311 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-20T15:05:42.312 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2022-03-20 15:05:40.105575899 +0000 2022-03-20T15:05:42.312 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2022-03-20 15:05:40.105575899 +0000 2022-03-20T15:05:42.312 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2022-03-20 15:05:40.105575899 +0000 2022-03-20T15:05:42.312 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: - 2022-03-20T15:05:42.318 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2022-03-20T15:05:42.318 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2022-03-20T15:05:42.318 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000278307 s, 1.8 MB/s 2022-03-20T15:05:42.322 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-20T15:05:42.329 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-20T15:05:42.330 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-20T15:05:42.330 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 519 Links: 1 2022-03-20T15:05:42.330 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-20T15:05:42.330 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2022-03-20 15:05:40.181575898 +0000 2022-03-20T15:05:42.331 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2022-03-20 15:05:40.181575898 +0000 2022-03-20T15:05:42.331 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2022-03-20 15:05:40.181575898 +0000 2022-03-20T15:05:42.331 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: - 2022-03-20T15:05:42.336 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2022-03-20T15:05:42.336 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2022-03-20T15:05:42.337 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000243917 s, 2.1 MB/s 2022-03-20T15:05:42.340 DEBUG:teuthology.orchestra.run.smithi045:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-20T15:05:42.464 INFO:teuthology.orchestra.run.smithi045.stdout:loop 2022-03-20T15:05:42.465 INFO:tasks.nvme_loop:Connecting nvme_loop smithi045:/dev/vg_nvme/lv_1... 2022-03-20T15:05:42.466 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-20T15:05:42.488 INFO:teuthology.orchestra.run.smithi045.stdout:1 2022-03-20T15:05:42.510 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/vg_nvme/lv_11 2022-03-20T15:05:42.533 INFO:tasks.nvme_loop:Connecting nvme_loop smithi045:/dev/vg_nvme/lv_2... 2022-03-20T15:05:42.533 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-20T15:05:42.556 INFO:teuthology.orchestra.run.smithi045.stdout:1 2022-03-20T15:05:42.579 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/vg_nvme/lv_21 2022-03-20T15:05:42.603 INFO:tasks.nvme_loop:Connecting nvme_loop smithi045:/dev/vg_nvme/lv_3... 2022-03-20T15:05:42.603 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-20T15:05:42.664 INFO:teuthology.orchestra.run.smithi045.stdout:1 2022-03-20T15:05:42.691 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/vg_nvme/lv_31 2022-03-20T15:05:42.713 INFO:tasks.nvme_loop:Connecting nvme_loop smithi045:/dev/vg_nvme/lv_4... 2022-03-20T15:05:42.713 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-20T15:05:42.735 INFO:teuthology.orchestra.run.smithi045.stdout:1 2022-03-20T15:05:42.762 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/vg_nvme/lv_41 2022-03-20T15:05:42.783 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2022-03-20T15:05:42.784 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/scratch_devs of=/dev/stdout 2022-03-20T15:05:42.789 DEBUG:teuthology.orchestra.run.smithi045:> sudo nvme list 2022-03-20T15:05:42.848 INFO:teuthology.orchestra.run.smithi045.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-20T15:05:42.849 INFO:teuthology.orchestra.run.smithi045.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-20T15:05:42.849 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/nvme0n1 PHMB75120098480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2022-03-20T15:05:42.849 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/nvme1n1 fd6ea8cd5c9845e7 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-20T15:05:42.849 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/nvme2n1 b82c0980ee057972 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-20T15:05:42.849 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/nvme3n1 add314352aaf4267 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-20T15:05:42.850 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/nvme4n1 b632dff1d45e1194 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-20T15:05:42.851 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-20T15:05:42.851 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2022-03-20T15:05:42.851 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/scratch_devs 2022-03-20T15:05:42.865 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2022-03-20T15:05:42.866 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/scratch_devs of=/dev/stdout 2022-03-20T15:05:42.872 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-20T15:05:42.872 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-20T15:05:42.920 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-20T15:05:42.920 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-20T15:05:42.921 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 5h/5d Inode: 499 Links: 1 2022-03-20T15:05:42.921 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-20T15:05:42.921 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2022-03-20 15:05:40.264097907 +0000 2022-03-20T15:05:42.921 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2022-03-20 15:05:40.264097907 +0000 2022-03-20T15:05:42.922 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2022-03-20 15:05:40.264097907 +0000 2022-03-20T15:05:42.922 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: - 2022-03-20T15:05:42.930 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2022-03-20T15:05:42.931 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2022-03-20T15:05:42.931 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000573434 s, 893 kB/s 2022-03-20T15:05:42.935 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-20T15:05:42.940 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-20T15:05:42.941 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-20T15:05:42.941 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 5h/5d Inode: 508 Links: 1 2022-03-20T15:05:42.941 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-20T15:05:42.941 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2022-03-20 15:05:40.292097907 +0000 2022-03-20T15:05:42.942 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2022-03-20 15:05:40.292097907 +0000 2022-03-20T15:05:42.942 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2022-03-20 15:05:40.292097907 +0000 2022-03-20T15:05:42.942 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: - 2022-03-20T15:05:42.947 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2022-03-20T15:05:42.948 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2022-03-20T15:05:42.948 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000464956 s, 1.1 MB/s 2022-03-20T15:05:42.951 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-20T15:05:42.957 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-20T15:05:42.957 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-20T15:05:42.957 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 5h/5d Inode: 513 Links: 1 2022-03-20T15:05:42.957 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-20T15:05:42.958 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2022-03-20 15:05:40.324097906 +0000 2022-03-20T15:05:42.958 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2022-03-20 15:05:40.324097906 +0000 2022-03-20T15:05:42.958 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2022-03-20 15:05:40.324097906 +0000 2022-03-20T15:05:42.958 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: - 2022-03-20T15:05:42.963 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2022-03-20T15:05:42.964 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2022-03-20T15:05:42.964 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.0004631 s, 1.1 MB/s 2022-03-20T15:05:42.968 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-20T15:05:43.017 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-20T15:05:43.018 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-20T15:05:43.018 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 5h/5d Inode: 519 Links: 1 2022-03-20T15:05:43.018 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-20T15:05:43.018 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2022-03-20 15:05:40.352097905 +0000 2022-03-20T15:05:43.019 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2022-03-20 15:05:40.352097905 +0000 2022-03-20T15:05:43.019 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2022-03-20 15:05:40.352097905 +0000 2022-03-20T15:05:43.019 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: - 2022-03-20T15:05:43.025 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2022-03-20T15:05:43.025 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2022-03-20T15:05:43.025 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000456668 s, 1.1 MB/s 2022-03-20T15:05:43.029 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-20T15:05:43.113 INFO:teuthology.orchestra.run.smithi123.stdout:loop 2022-03-20T15:05:43.114 INFO:tasks.nvme_loop:Connecting nvme_loop smithi123:/dev/vg_nvme/lv_1... 2022-03-20T15:05:43.115 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-20T15:05:43.136 INFO:teuthology.orchestra.run.smithi123.stdout:1 2022-03-20T15:05:43.159 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/vg_nvme/lv_11 2022-03-20T15:05:43.182 INFO:tasks.nvme_loop:Connecting nvme_loop smithi123:/dev/vg_nvme/lv_2... 2022-03-20T15:05:43.183 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-20T15:05:43.202 INFO:teuthology.orchestra.run.smithi123.stdout:1 2022-03-20T15:05:43.224 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/vg_nvme/lv_21 2022-03-20T15:05:43.246 INFO:tasks.nvme_loop:Connecting nvme_loop smithi123:/dev/vg_nvme/lv_3... 2022-03-20T15:05:43.246 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-20T15:05:43.310 INFO:teuthology.orchestra.run.smithi123.stdout:1 2022-03-20T15:05:43.337 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/vg_nvme/lv_31 2022-03-20T15:05:43.356 INFO:tasks.nvme_loop:Connecting nvme_loop smithi123:/dev/vg_nvme/lv_4... 2022-03-20T15:05:43.356 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-20T15:05:43.376 INFO:teuthology.orchestra.run.smithi123.stdout:1 2022-03-20T15:05:43.399 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/vg_nvme/lv_41 2022-03-20T15:05:43.418 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2022-03-20T15:05:43.419 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/scratch_devs of=/dev/stdout 2022-03-20T15:05:43.424 DEBUG:teuthology.orchestra.run.smithi123:> sudo nvme list 2022-03-20T15:05:43.482 INFO:teuthology.orchestra.run.smithi123.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-20T15:05:43.482 INFO:teuthology.orchestra.run.smithi123.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-20T15:05:43.482 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/nvme0n1 PHFT620400YK400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV10171 2022-03-20T15:05:43.483 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/nvme1n1 a27b741560a4a41c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-20T15:05:43.483 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/nvme2n1 0815bd991cb8b863 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-20T15:05:43.483 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/nvme3n1 2214e05017fe828c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-20T15:05:43.483 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/nvme4n1 bea5c30da342663f Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-20T15:05:43.484 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-20T15:05:43.485 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2022-03-20T15:05:43.485 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/scratch_devs 2022-03-20T15:05:43.499 INFO:teuthology.run_tasks:Running task rook... 2022-03-20T15:05:43.568 INFO:tasks.rook:Rook start 2022-03-20T15:05:43.568 INFO:tasks.rook:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'e92561bf8a1bc96ca490936c6f46ac4673e1b26e', 'rook_image': 'rook/ceph:master'} 2022-03-20T15:05:43.569 INFO:tasks.rook:Ceph image is quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e 2022-03-20T15:05:43.569 INFO:tasks.rook:Cloning https://github.com/rook/rook branch master 2022-03-20T15:05:43.569 DEBUG:teuthology.orchestra.run.smithi006:> rm -rf rook && git clone --single-branch --branch master https://github.com/rook/rook rook 2022-03-20T15:05:43.579 INFO:teuthology.orchestra.run.smithi006.stderr:Cloning into 'rook'... 2022-03-20T15:05:46.003 INFO:tasks.rook:/home/teuthworker 2022-03-20T15:05:46.004 DEBUG:teuthology.orchestra.run.smithi006:> PATH=/sbin:/usr/sbin ip addr show 2022-03-20T15:05:46.011 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-03-20T15:05:46.011 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-03-20T15:05:46.012 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2022-03-20T15:05:46.012 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2022-03-20T15:05:46.012 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2022-03-20T15:05:46.012 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2022-03-20T15:05:46.013 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-20T15:05:46.013 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2022-03-20T15:05:46.013 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2022-03-20T15:05:46.013 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-20T15:05:46.014 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2022-03-20T15:05:46.014 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2022-03-20T15:05:46.014 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2022-03-20T15:05:46.014 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2022-03-20T15:05:46.015 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2022-03-20T15:05:46.015 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic ens1f0 2022-03-20T15:05:46.015 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42838sec preferred_lft 42838sec 2022-03-20T15:05:46.015 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link 2022-03-20T15:05:46.015 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2022-03-20T15:05:46.016 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-20T15:05:46.016 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2022-03-20T15:05:46.016 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2022-03-20T15:05:46.016 INFO:teuthology.orchestra.run.smithi006.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2022-03-20T15:05:46.016 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 02:42:25:ec:2e:b8 brd ff:ff:ff:ff:ff:ff 2022-03-20T15:05:46.017 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2022-03-20T15:05:46.017 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2022-03-20T15:05:46.018 DEBUG:teuthology.orchestra.run.smithi006:> which systemctl 2022-03-20T15:05:46.025 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/systemctl 2022-03-20T15:05:46.025 DEBUG:teuthology.orchestra.run.smithi006:> true 2022-03-20T15:05:46.097 DEBUG:teuthology.orchestra.run.smithi006:> sudo lsb_release -is 2022-03-20T15:05:46.145 INFO:teuthology.orchestra.run.smithi006.stdout:Ubuntu 2022-03-20T15:05:46.151 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-20T15:05:46.151 INFO:tasks.rook:['__class__', '__del__', '__delattr__', '__dir__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_format_size', '_reconnect', '_runner', '_set_iface_and_cidr', '_sftp_get_file', '_sftp_get_size', '_sftp_open_file', '_sftp_put_file', 'chcon', 'chmod', 'connect', 'copy_file', 'ensure_online', 'get_file', 'get_tar', 'get_tar_stream', 'mkdtemp', 'mktemp', 'move_file', 'put_file', 'read_file', 'reconnect', 'remove', 'run', 'sh', 'sh_file', 'sudo_write_file', 'write_file'] 2022-03-20T15:05:46.161 DEBUG:teuthology.orchestra.remote:smithi006:rook/deploy/examples/operator.yaml is 20KB 2022-03-20T15:05:46.175 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2022-03-20T15:05:46.175 DEBUG:teuthology.orchestra.run.smithi006:> dd if=rook/deploy/examples/operator.yaml of=/dev/stdout 2022-03-20T15:05:46.185 INFO:tasks.rook:Patching operator to use image rook/ceph:master 2022-03-20T15:05:46.222 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2022-03-20T15:05:46.223 DEBUG:teuthology.orchestra.run.smithi006:> dd of=operator.yaml 2022-03-20T15:05:46.235 INFO:tasks.rook:Deploying operator 2022-03-20T15:05:46.244 DEBUG:teuthology.orchestra.remote:smithi006:rook/deploy/examples/operator.yaml is 20KB 2022-03-20T15:05:46.271 DEBUG:teuthology.orchestra.remote:smithi006:rook/deploy/examples/operator.yaml is 20KB 2022-03-20T15:05:46.285 DEBUG:teuthology.orchestra.run.smithi006:> kubectl create -f rook/deploy/examples/crds.yaml -f rook/deploy/examples/common.yaml -f operator.yaml 2022-03-20T15:05:46.547 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created 2022-03-20T15:05:46.574 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/cephbucketnotifications.ceph.rook.io created 2022-03-20T15:05:46.599 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/cephbuckettopics.ceph.rook.io created 2022-03-20T15:05:46.623 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/cephclients.ceph.rook.io created 2022-03-20T15:05:46.732 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created 2022-03-20T15:05:46.759 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemmirrors.ceph.rook.io created 2022-03-20T15:05:46.867 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created 2022-03-20T15:05:46.882 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemsubvolumegroups.ceph.rook.io created 2022-03-20T15:05:46.939 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created 2022-03-20T15:05:47.049 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectrealms.ceph.rook.io created 2022-03-20T15:05:47.129 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created 2022-03-20T15:05:47.192 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created 2022-03-20T15:05:47.207 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzonegroups.ceph.rook.io created 2022-03-20T15:05:47.233 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzones.ceph.rook.io created 2022-03-20T15:05:47.261 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/cephrbdmirrors.ceph.rook.io created 2022-03-20T15:05:47.349 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/objectbucketclaims.objectbucket.io created 2022-03-20T15:05:47.359 INFO:teuthology.orchestra.run.smithi006.stdout:customresourcedefinition.apiextensions.k8s.io/objectbuckets.objectbucket.io created 2022-03-20T15:05:47.407 INFO:teuthology.orchestra.run.smithi006.stdout:namespace/rook-ceph created 2022-03-20T15:05:47.432 INFO:teuthology.orchestra.run.smithi006.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-20T15:05:47.516 INFO:teuthology.orchestra.run.smithi006.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-external-provisioner-runner created 2022-03-20T15:05:47.607 INFO:teuthology.orchestra.run.smithi006.stdout:clusterrole.rbac.authorization.k8s.io/psp:rook created 2022-03-20T15:05:47.607 INFO:teuthology.orchestra.run.smithi006.stdout:clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-20T15:05:47.607 INFO:teuthology.orchestra.run.smithi006.stdout:clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created 2022-03-20T15:05:47.607 INFO:teuthology.orchestra.run.smithi006.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-20T15:05:47.608 INFO:teuthology.orchestra.run.smithi006.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-20T15:05:47.608 INFO:teuthology.orchestra.run.smithi006.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-20T15:05:47.608 INFO:teuthology.orchestra.run.smithi006.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-20T15:05:47.613 INFO:teuthology.orchestra.run.smithi006.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-20T15:05:47.685 INFO:teuthology.orchestra.run.smithi006.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-20T15:05:47.689 INFO:teuthology.orchestra.run.smithi006.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-20T15:05:47.701 INFO:teuthology.orchestra.run.smithi006.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-20T15:05:47.709 INFO:teuthology.orchestra.run.smithi006.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role created 2022-03-20T15:05:47.717 INFO:teuthology.orchestra.run.smithi006.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-20T15:05:47.726 INFO:teuthology.orchestra.run.smithi006.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created 2022-03-20T15:05:47.734 INFO:teuthology.orchestra.run.smithi006.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-20T15:05:47.758 INFO:teuthology.orchestra.run.smithi006.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-20T15:05:47.768 INFO:teuthology.orchestra.run.smithi006.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-20T15:05:47.776 INFO:teuthology.orchestra.run.smithi006.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-20T15:05:47.811 INFO:teuthology.orchestra.run.smithi006.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-20T15:05:47.814 INFO:teuthology.orchestra.run.smithi006.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system-psp created 2022-03-20T15:05:47.841 INFO:teuthology.orchestra.run.smithi006.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-plugin-sa-psp created 2022-03-20T15:05:47.851 INFO:teuthology.orchestra.run.smithi006.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-provisioner-sa-psp created 2022-03-20T15:05:47.859 INFO:teuthology.orchestra.run.smithi006.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-plugin-sa-psp created 2022-03-20T15:05:47.867 INFO:teuthology.orchestra.run.smithi006.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-provisioner-sa-psp created 2022-03-20T15:05:47.876 INFO:teuthology.orchestra.run.smithi006.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-20T15:05:47.876 INFO:teuthology.orchestra.run.smithi006.stdout:podsecuritypolicy.policy/00-rook-privileged created 2022-03-20T15:05:47.884 INFO:teuthology.orchestra.run.smithi006.stdout:role.rbac.authorization.k8s.io/cephfs-external-provisioner-cfg created 2022-03-20T15:05:47.907 INFO:teuthology.orchestra.run.smithi006.stdout:role.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-20T15:05:47.944 INFO:teuthology.orchestra.run.smithi006.stdout:role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created 2022-03-20T15:05:47.947 INFO:teuthology.orchestra.run.smithi006.stdout:role.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-20T15:05:47.968 INFO:teuthology.orchestra.run.smithi006.stdout:role.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-20T15:05:47.976 INFO:teuthology.orchestra.run.smithi006.stdout:role.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-20T15:05:47.984 INFO:teuthology.orchestra.run.smithi006.stdout:role.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-20T15:05:48.008 INFO:teuthology.orchestra.run.smithi006.stdout:role.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-20T15:05:48.017 INFO:teuthology.orchestra.run.smithi006.stdout:rolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role-cfg created 2022-03-20T15:05:48.026 INFO:teuthology.orchestra.run.smithi006.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin-role-cfg created 2022-03-20T15:05:48.034 INFO:teuthology.orchestra.run.smithi006.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created 2022-03-20T15:05:48.042 INFO:teuthology.orchestra.run.smithi006.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-20T15:05:48.083 INFO:teuthology.orchestra.run.smithi006.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-20T15:05:48.092 INFO:teuthology.orchestra.run.smithi006.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter-psp created 2022-03-20T15:05:48.101 INFO:teuthology.orchestra.run.smithi006.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-default-psp created 2022-03-20T15:05:48.109 INFO:teuthology.orchestra.run.smithi006.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-20T15:05:48.117 INFO:teuthology.orchestra.run.smithi006.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-psp created 2022-03-20T15:05:48.126 INFO:teuthology.orchestra.run.smithi006.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-20T15:05:48.134 INFO:teuthology.orchestra.run.smithi006.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-20T15:05:48.158 INFO:teuthology.orchestra.run.smithi006.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd-psp created 2022-03-20T15:05:48.176 INFO:teuthology.orchestra.run.smithi006.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-20T15:05:48.211 INFO:teuthology.orchestra.run.smithi006.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-20T15:05:48.214 INFO:teuthology.orchestra.run.smithi006.stdout:serviceaccount/rook-ceph-cmd-reporter created 2022-03-20T15:05:48.226 INFO:teuthology.orchestra.run.smithi006.stdout:serviceaccount/rook-ceph-mgr created 2022-03-20T15:05:48.234 INFO:teuthology.orchestra.run.smithi006.stdout:serviceaccount/rook-ceph-osd created 2022-03-20T15:05:48.259 INFO:teuthology.orchestra.run.smithi006.stdout:serviceaccount/rook-ceph-purge-osd created 2022-03-20T15:05:48.292 INFO:teuthology.orchestra.run.smithi006.stdout:serviceaccount/rook-ceph-system created 2022-03-20T15:05:48.367 INFO:teuthology.orchestra.run.smithi006.stdout:serviceaccount/rook-csi-cephfs-plugin-sa created 2022-03-20T15:05:48.392 INFO:teuthology.orchestra.run.smithi006.stdout:serviceaccount/rook-csi-cephfs-provisioner-sa created 2022-03-20T15:05:48.418 INFO:teuthology.orchestra.run.smithi006.stdout:serviceaccount/rook-csi-rbd-plugin-sa created 2022-03-20T15:05:48.443 INFO:teuthology.orchestra.run.smithi006.stdout:serviceaccount/rook-csi-rbd-provisioner-sa created 2022-03-20T15:05:48.511 INFO:teuthology.orchestra.run.smithi006.stdout:configmap/rook-ceph-operator-config created 2022-03-20T15:05:48.535 INFO:teuthology.orchestra.run.smithi006.stdout:deployment.apps/rook-ceph-operator created 2022-03-20T15:05:48.540 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-20T15:05:48.686 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:05:48.686 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-operator-75dd789779-md7p5 0/1 Pending 0 0s 2022-03-20T15:05:58.689 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-20T15:05:58.753 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:05:58.753 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-operator-75dd789779-md7p5 0/1 Pending 0 10s 2022-03-20T15:06:08.756 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-20T15:06:08.913 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:06:08.913 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-operator-75dd789779-md7p5 0/1 Pending 0 20s 2022-03-20T15:06:18.916 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-20T15:06:19.063 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:06:19.063 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-operator-75dd789779-md7p5 0/1 Pending 0 31s 2022-03-20T15:06:29.070 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-20T15:06:29.136 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:06:29.136 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-operator-75dd789779-md7p5 0/1 ContainerCreating 0 41s 2022-03-20T15:06:39.140 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-20T15:06:39.203 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:06:39.203 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-operator-75dd789779-md7p5 0/1 ContainerCreating 0 51s 2022-03-20T15:06:49.206 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-20T15:06:49.275 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:06:49.276 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-operator-75dd789779-md7p5 0/1 ContainerCreating 0 61s 2022-03-20T15:06:59.279 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-20T15:06:59.345 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:06:59.346 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-operator-75dd789779-md7p5 1/1 Running 0 71s 2022-03-20T15:06:59.348 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph logs -f rook-ceph-operator-75dd789779-md7p5 2022-03-20T15:06:59.356 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2022-03-20T15:06:59.356 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/scratch_devs of=/dev/stdout 2022-03-20T15:06:59.364 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2022-03-20T15:06:59.364 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/scratch_devs of=/dev/stdout 2022-03-20T15:06:59.372 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2022-03-20T15:06:59.373 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/scratch_devs of=/dev/stdout 2022-03-20T15:06:59.382 INFO:tasks.rook: override: [mgr] debug mgr = 20 2022-03-20T15:06:59.383 INFO:tasks.rook: override: [mgr] debug ms = 1 2022-03-20T15:06:59.383 INFO:tasks.rook: override: [mon] debug mon = 20 2022-03-20T15:06:59.383 INFO:tasks.rook: override: [mon] debug ms = 1 2022-03-20T15:06:59.384 INFO:tasks.rook: override: [mon] debug paxos = 20 2022-03-20T15:06:59.384 INFO:tasks.rook: override: [osd] debug ms = 1 2022-03-20T15:06:59.384 INFO:tasks.rook: override: [osd] debug osd = 20 2022-03-20T15:06:59.385 INFO:tasks.rook:Config: b"[global]\n\nlog to file = true\n\nmon clock drift allowed = 1.000\n\n# replicate across OSDs, not hosts\nosd crush chooseleaf type = 0\n\n# enable some debugging\nauth debug = true\nms die on old message = true\nms die on bug = true\ndebug asserts on shutdown = true\n\n\n[osd]\n# debugging\nosd debug shutdown = true\nosd debug op order = true\nosd debug verify stray on activate = true\nosd debug pg log writeout = true\nosd debug verify cached snaps = true\nosd debug verify missing on start = true\nosd debug misdirected ops = true\nosd op queue = debug_random\nosd op queue cut off = debug_random\nosd shutdown pgref assert = true\nbdev debug aio = true\nosd sloppy crc = true\ndebug ms = 1\ndebug osd = 20\n\n\n[mon]\n# rotate auth tickets quickly to exercise renewal paths\nauth mon ticket ttl = 660# 11m\nauth service ticket ttl = 240# 4m\n\n# don't complain about global id reclaim\nmon_warn_on_insecure_global_id_reclaim = false\nmon_warn_on_insecure_global_id_reclaim_allowed = false\ndebug mon = 20\ndebug ms = 1\ndebug paxos = 20\n[mgr]\ndebug mgr = 20\ndebug ms = 1\n" 2022-03-20T15:06:59.387 DEBUG:teuthology.orchestra.run.smithi006:> kubectl create -f - 2022-03-20T15:06:59.452 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:55.785607 I | rookcmd: starting Rook v1.8.0-alpha.0.441.g2b2469cf1 with arguments '/usr/local/bin/rook ceph operator' 2022-03-20T15:06:59.452 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:55.785719 I | rookcmd: flag values: --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-level=INFO, --operator-image=, --service-account= 2022-03-20T15:06:59.452 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:55.785722 I | cephcmd: starting Rook-Ceph operator 2022-03-20T15:06:59.453 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:55.930525 I | cephcmd: base ceph version inside the rook operator image is "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)" 2022-03-20T15:06:59.453 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:55.939846 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-20T15:06:59.453 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:55.939867 I | operator: watching all namespaces for Ceph CRs 2022-03-20T15:06:59.453 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:55.939914 I | operator: setting up schemes 2022-03-20T15:06:59.454 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:55.941489 I | operator: setting up the controller-runtime manager 2022-03-20T15:06:59.454 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:56.246660 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-20T15:06:59.454 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:56.250071 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-20T15:06:59.454 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:56.250126 I | ceph-cluster-controller: successfully started 2022-03-20T15:06:59.455 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:56.250237 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-20T15:06:59.455 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:56.250270 I | ceph-crashcollector-controller: successfully started 2022-03-20T15:06:59.455 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:56.250321 I | ceph-block-pool-controller: successfully started 2022-03-20T15:06:59.455 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:56.250394 I | ceph-object-store-user-controller: successfully started 2022-03-20T15:06:59.455 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:56.250461 I | ceph-object-realm-controller: successfully started 2022-03-20T15:06:59.456 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:56.250494 I | ceph-object-zonegroup-controller: successfully started 2022-03-20T15:06:59.456 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:56.250525 I | ceph-object-zone-controller: successfully started 2022-03-20T15:06:59.456 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:56.250706 I | ceph-object-controller: successfully started 2022-03-20T15:06:59.456 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:56.250765 I | ceph-file-controller: successfully started 2022-03-20T15:06:59.457 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:56.250831 I | ceph-nfs-controller: successfully started 2022-03-20T15:06:59.457 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:56.250893 I | ceph-rbd-mirror-controller: successfully started 2022-03-20T15:06:59.457 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:56.250945 I | ceph-client-controller: successfully started 2022-03-20T15:06:59.457 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:56.250986 I | ceph-filesystem-mirror-controller: successfully started 2022-03-20T15:06:59.458 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:56.251046 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-20T15:06:59.458 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:56.251081 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-20T15:06:59.458 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:56.251116 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-20T15:06:59.458 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:56.251152 I | ceph-bucket-topic: successfully started 2022-03-20T15:06:59.458 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:56.251172 I | ceph-bucket-notification: successfully started 2022-03-20T15:06:59.459 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:56.251189 I | ceph-bucket-notification: successfully started 2022-03-20T15:06:59.459 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:56.251208 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-20T15:06:59.459 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:56.252576 I | operator: starting the controller-runtime manager 2022-03-20T15:06:59.459 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:56.356664 I | ceph-csi: CSI Ceph RBD driver disabled 2022-03-20T15:06:59.460 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:56.356700 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists 2022-03-20T15:06:59.460 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:56.360520 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists 2022-03-20T15:06:59.460 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:56.361904 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-20T15:06:59.460 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:56.361924 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-20T15:06:59.461 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:56.361933 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-20T15:06:59.461 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:56.364443 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-20T15:06:59.461 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:56.368136 I | ceph-csi: successfully removed CSI Ceph RBD driver 2022-03-20T15:06:59.461 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:56.368158 I | ceph-csi: CSI CephFS driver disabled 2022-03-20T15:06:59.461 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:56.368168 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists 2022-03-20T15:06:59.462 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:56.370193 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists 2022-03-20T15:06:59.462 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:06:56.376182 I | ceph-csi: successfully removed CSI CephFS driver 2022-03-20T15:07:00.399 INFO:teuthology.orchestra.run.smithi006.stdout:configmap/rook-config-override created 2022-03-20T15:07:00.403 INFO:tasks.rook:Cluster: apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: allowUnsupported: true image: quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e dataDirHostPath: /var/lib/rook mgr: count: 1 modules: - enabled: true name: rook mon: allowMultiplePerNode: true count: 3 skipUpgradeChecks: true 2022-03-20T15:07:00.403 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2022-03-20T15:07:00.404 DEBUG:teuthology.orchestra.run.smithi006:> dd of=cluster.yaml 2022-03-20T15:07:00.410 DEBUG:teuthology.orchestra.run.smithi006:> kubectl create -f cluster.yaml 2022-03-20T15:07:02.791 INFO:teuthology.orchestra.run.smithi006.stdout:cephcluster.ceph.rook.io/rook-ceph created 2022-03-20T15:07:02.797 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:07:02.800834 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "rook-ceph" 2022-03-20T15:07:02.805 DEBUG:teuthology.orchestra.remote:smithi006:rook/deploy/examples/operator.yaml is 20KB 2022-03-20T15:07:02.808 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:07:02.811553 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-20T15:07:02.816 DEBUG:teuthology.orchestra.run.smithi006:> kubectl create -f rook/deploy/examples/toolbox.yaml 2022-03-20T15:07:02.832 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:07:02.835947 I | ceph-cluster-controller: clusterInfo not yet found, must be a new cluster. 2022-03-20T15:07:02.832 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:07:02.836079 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-20T15:07:02.859 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:07:02.862710 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e... 2022-03-20T15:07:02.887 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:07:02.889117 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-20T15:07:02.887 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:07:02.889149 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-20T15:07:02.887 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:07:02.889157 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-20T15:07:02.888 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:07:02.889162 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-20T15:07:02.888 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:07:02.889167 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-20T15:07:02.888 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:07:02.889176 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-20T15:07:02.888 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:07:02.889182 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-20T15:07:02.889 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:07:02.889188 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-20T15:07:02.889 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:07:02.889194 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-20T15:07:02.889 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:07:02.889199 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-20T15:07:02.889 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:07:02.889205 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-20T15:07:02.890 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:07:02.889211 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-20T15:07:02.890 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:07:02.889217 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-20T15:07:02.890 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:07:02.889222 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-20T15:07:02.890 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:07:02.889226 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-20T15:07:02.891 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:07:02.889233 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-20T15:07:02.891 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:07:02.889291 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-20T15:07:02.891 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:07:02.889305 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-20T15:07:03.158 INFO:teuthology.orchestra.run.smithi006.stdout:deployment.apps/rook-ceph-tools created 2022-03-20T15:07:03.163 INFO:tasks.rook:Waiting for tools container to start 2022-03-20T15:07:03.163 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T15:07:03.271 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:07:03.271 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-tools-d6d7c985c-54sh5 0/1 Pending 0 0s 2022-03-20T15:07:08.273 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T15:07:08.350 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:07:08.350 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-tools-d6d7c985c-54sh5 0/1 ContainerCreating 0 5s 2022-03-20T15:07:13.352 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T15:07:13.415 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:07:13.415 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-tools-d6d7c985c-54sh5 0/1 ContainerCreating 0 10s 2022-03-20T15:07:18.418 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T15:07:18.484 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:07:18.485 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-tools-d6d7c985c-54sh5 0/1 ContainerCreating 0 15s 2022-03-20T15:07:23.488 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T15:07:23.552 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:07:23.553 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-tools-d6d7c985c-54sh5 0/1 ContainerCreating 0 20s 2022-03-20T15:07:28.556 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T15:07:28.623 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:07:28.624 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-tools-d6d7c985c-54sh5 0/1 ContainerCreating 0 25s 2022-03-20T15:07:33.626 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T15:07:33.687 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:07:33.688 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-tools-d6d7c985c-54sh5 0/1 ContainerCreating 0 30s 2022-03-20T15:07:38.691 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T15:07:38.763 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:07:38.763 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-tools-d6d7c985c-54sh5 0/1 ContainerCreating 0 35s 2022-03-20T15:07:43.766 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T15:07:43.845 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:07:43.845 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-tools-d6d7c985c-54sh5 0/1 ContainerCreating 0 40s 2022-03-20T15:07:48.847 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T15:07:48.920 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:07:48.921 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-tools-d6d7c985c-54sh5 0/1 ContainerCreating 0 45s 2022-03-20T15:07:53.924 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T15:07:53.992 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:07:53.992 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-tools-d6d7c985c-54sh5 0/1 ContainerCreating 0 50s 2022-03-20T15:07:58.995 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T15:07:59.066 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:07:59.066 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-tools-d6d7c985c-54sh5 0/1 ContainerCreating 0 56s 2022-03-20T15:08:04.070 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T15:08:04.136 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:08:04.137 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-tools-d6d7c985c-54sh5 0/1 ContainerCreating 0 61s 2022-03-20T15:08:09.139 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T15:08:09.198 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:08:09.199 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-tools-d6d7c985c-54sh5 0/1 ContainerCreating 0 66s 2022-03-20T15:08:14.202 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T15:08:14.269 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:08:14.269 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-tools-d6d7c985c-54sh5 0/1 ContainerCreating 0 71s 2022-03-20T15:08:19.273 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T15:08:19.347 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:08:19.347 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-tools-d6d7c985c-54sh5 0/1 ContainerCreating 0 76s 2022-03-20T15:08:24.350 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T15:08:24.420 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:08:24.420 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-tools-d6d7c985c-54sh5 0/1 ContainerCreating 0 81s 2022-03-20T15:08:29.423 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T15:08:29.506 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:08:29.507 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-tools-d6d7c985c-54sh5 0/1 ContainerCreating 0 86s 2022-03-20T15:08:34.510 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T15:08:34.575 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:08:34.575 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-tools-d6d7c985c-54sh5 0/1 ContainerCreating 0 91s 2022-03-20T15:08:39.577 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T15:08:39.650 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:08:39.651 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-tools-d6d7c985c-54sh5 0/1 ContainerCreating 0 96s 2022-03-20T15:08:41.991 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:41.990541 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-20T15:08:41.991 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:41.990584 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-20T15:08:41.992 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:41.990595 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-20T15:08:41.992 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:41.990601 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-20T15:08:41.992 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:41.990608 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-20T15:08:41.992 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:41.990613 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-20T15:08:41.993 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:41.990619 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-20T15:08:41.993 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:41.990625 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-20T15:08:41.993 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:41.990632 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-20T15:08:41.993 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:41.990638 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-20T15:08:41.993 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:41.990645 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-20T15:08:41.994 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:41.990650 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-20T15:08:41.994 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:41.990658 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-20T15:08:41.995 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:41.990671 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-20T15:08:41.996 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:41.990681 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-20T15:08:41.996 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:41.990688 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-20T15:08:41.996 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:41.990694 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-20T15:08:41.996 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:41.990700 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-20T15:08:41.997 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:41.990706 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-20T15:08:41.997 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:41.990712 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-20T15:08:41.997 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:41.990719 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-20T15:08:41.997 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:41.990727 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-20T15:08:41.998 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:41.990733 I | ceph-csi: Kubernetes version is 1.23 2022-03-20T15:08:41.998 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:41.990743 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-20T15:08:41.998 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:41.990753 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-20T15:08:42.017 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:42.017911 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-20T15:08:42.028 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:42.028232 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-20T15:08:42.028 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:42.028256 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-20T15:08:42.028 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:42.028261 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-20T15:08:42.028 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:42.028265 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-20T15:08:42.029 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:42.028269 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-20T15:08:42.029 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:42.028273 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-20T15:08:42.029 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:42.028276 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-20T15:08:42.063 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:42.063941 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-20T15:08:42.063 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:42.063968 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-20T15:08:42.064 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:42.063975 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-20T15:08:42.199 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:42.199759 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-20T15:08:42.330 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:42.330608 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-20T15:08:42.330 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:42.330634 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-20T15:08:42.331 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:42.330639 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-20T15:08:42.393 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:42.393594 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-20T15:08:42.393 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:42.393622 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-20T15:08:42.393 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:42.393628 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-20T15:08:42.537 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:42.537400 I | ceph-csi: successfully started CSI CephFS driver 2022-03-20T15:08:42.954 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:42.954640 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-20T15:08:43.166 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:43.166548 I | ceph-csi: CSIDriver object created for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-20T15:08:43.166 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:43.166580 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-20T15:08:43.434 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:43.434628 I | ceph-csi: CSIDriver object created for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-20T15:08:44.654 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T15:08:44.722 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:08:44.723 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-tools-d6d7c985c-54sh5 0/1 ContainerCreating 0 101s 2022-03-20T15:08:49.726 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T15:08:49.797 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:08:49.798 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-tools-d6d7c985c-54sh5 0/1 ContainerCreating 0 106s 2022-03-20T15:08:53.071 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:53.072335 I | ceph-spec: detected ceph image version: "17.0.0-11187 quincy" 2022-03-20T15:08:53.072 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:53.072361 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-20T15:08:53.122 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:53.123199 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11187 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e" 2022-03-20T15:08:53.215 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:53.216045 I | op-mon: start running mons 2022-03-20T15:08:53.282 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:53.283259 I | op-mon: creating mon secrets for a new cluster 2022-03-20T15:08:53.321 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:53.321973 I | op-mon: existing maxMonID not found or failed to load. configmaps "rook-ceph-mon-endpoints" not found 2022-03-20T15:08:53.352 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:53.353171 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[]}] data: mapping:{"node":{}} maxMonId:-1] 2022-03-20T15:08:54.152 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:54.153061 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T15:08:54.153 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:54.153283 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T15:08:54.800 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T15:08:54.866 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:08:54.866 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-tools-d6d7c985c-54sh5 0/1 ContainerCreating 0 111s 2022-03-20T15:08:55.764 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:55.764853 I | op-mon: targeting the mon count 3 2022-03-20T15:08:55.770 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:55.771308 I | op-mon: created canary deployment rook-ceph-mon-a-canary 2022-03-20T15:08:55.791 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:55.792647 I | op-mon: created canary deployment rook-ceph-mon-b-canary 2022-03-20T15:08:55.827 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:55.819547 I | op-mon: created canary deployment rook-ceph-mon-c-canary 2022-03-20T15:08:56.554 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:56.554895 I | op-mon: canary monitor deployment rook-ceph-mon-a-canary scheduled to smithi045 2022-03-20T15:08:56.554 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:56.554927 I | op-mon: mon a assigned to node smithi045 2022-03-20T15:08:57.005 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:57.006596 I | op-mon: canary monitor deployment rook-ceph-mon-b-canary scheduled to smithi123 2022-03-20T15:08:57.006 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:57.006617 I | op-mon: mon b assigned to node smithi123 2022-03-20T15:08:57.095 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:57.096537 I | op-mon: canary monitor deployment rook-ceph-mon-c-canary scheduled to smithi006 2022-03-20T15:08:57.096 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:57.096566 I | op-mon: mon c assigned to node smithi006 2022-03-20T15:08:57.151 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:57.152276 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-a-canary" 2022-03-20T15:08:57.230 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:57.230821 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-b-canary" 2022-03-20T15:08:57.579 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:57.580227 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-c-canary" 2022-03-20T15:08:57.776 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:57.777974 I | op-mon: creating mon a 2022-03-20T15:08:58.026 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:58.026623 I | op-mon: mon "a" ip is 10.107.211.141 2022-03-20T15:08:58.298 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:58.298420 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-20T15:08:58.298 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:58.298525 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.211.141:6789"]}] data:a=10.107.211.141:6789 mapping:{"node":{"a":{"Name":"smithi045","Hostname":"smithi045","Address":"172.21.15.45"},"b":{"Name":"smithi123","Hostname":"smithi123","Address":"172.21.15.123"},"c":{"Name":"smithi006","Hostname":"smithi006","Address":"172.21.15.6"}}} maxMonId:-1] 2022-03-20T15:08:58.298 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:58.298743 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-20T15:08:58.447 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:58.447828 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T15:08:58.447 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:58.448080 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T15:08:58.764 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:58.765003 I | op-mon: 0 of 1 expected mons are ready. creating or updating deployments without checking quorum in attempt to achieve a healthy mon cluster 2022-03-20T15:08:58.988 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:58.989068 I | op-mon: updating maxMonID from -1 to 0 after committing mon "a" 2022-03-20T15:08:59.771 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:59.772484 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.211.141:6789"]}] data:a=10.107.211.141:6789 mapping:{"node":{"a":{"Name":"smithi045","Hostname":"smithi045","Address":"172.21.15.45"},"b":{"Name":"smithi123","Hostname":"smithi123","Address":"172.21.15.123"},"c":{"Name":"smithi006","Hostname":"smithi006","Address":"172.21.15.6"}}} maxMonId:0] 2022-03-20T15:08:59.772 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:59.772523 I | op-mon: waiting for mon quorum with [a] 2022-03-20T15:08:59.869 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T15:08:59.940 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:08:59.940 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-tools-d6d7c985c-54sh5 0/1 ContainerCreating 0 116s 2022-03-20T15:08:59.956 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:59.957778 I | op-mon: mon a is not yet running 2022-03-20T15:08:59.957 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:08:59.957809 I | op-mon: mons running: [] 2022-03-20T15:09:02.979 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:02.980697 I | op-mon: parsing mon endpoints: a=10.107.211.141:6789 2022-03-20T15:09:02.980 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:02.980757 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-20T15:09:02.980 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:02.980770 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-20T15:09:02.982 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:02.982355 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-20T15:09:02.982 INFO:tasks.rook.operator.smithi006.stdout:I0320 15:09:02.982426 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-20T15:09:04.943 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T15:09:05.013 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:09:05.014 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-tools-d6d7c985c-54sh5 0/1 ContainerCreating 0 2m2s 2022-03-20T15:09:10.017 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T15:09:10.093 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:09:10.093 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-tools-d6d7c985c-54sh5 0/1 ContainerCreating 0 2m7s 2022-03-20T15:09:15.096 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T15:09:15.176 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:09:15.177 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-tools-d6d7c985c-54sh5 0/1 ContainerCreating 0 2m12s 2022-03-20T15:09:20.128 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:20.130139 I | op-mon: mons running: [a] 2022-03-20T15:09:20.179 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T15:09:20.256 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:09:20.256 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-tools-d6d7c985c-54sh5 0/1 ContainerCreating 0 2m17s 2022-03-20T15:09:25.259 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T15:09:25.328 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:09:25.329 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-tools-d6d7c985c-54sh5 0/1 ContainerCreating 0 2m22s 2022-03-20T15:09:30.332 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T15:09:30.399 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:09:30.399 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-tools-d6d7c985c-54sh5 0/1 ContainerCreating 0 2m27s 2022-03-20T15:09:30.654 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:30.655909 I | op-mon: Monitors in quorum: [a] 2022-03-20T15:09:30.654 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:30.655935 I | op-mon: mons created: 1 2022-03-20T15:09:30.933 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:30.934929 I | op-mon: waiting for mon quorum with [a] 2022-03-20T15:09:31.005 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:31.006788 I | op-mon: mons running: [a] 2022-03-20T15:09:31.257 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:31.259236 I | op-mon: Monitors in quorum: [a] 2022-03-20T15:09:31.258 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:31.259265 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T15:09:31.593 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:31.595028 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T15:09:31.593 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:31.595046 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T15:09:31.927 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:31.928610 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T15:09:31.927 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:31.928629 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T15:09:32.187 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:32.189298 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T15:09:32.188 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:32.189318 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-20T15:09:32.481 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:32.483321 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-20T15:09:32.482 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:32.483352 I | op-config: deleting "log file" option from the mon configuration database 2022-03-20T15:09:32.696 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:32.697934 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-20T15:09:32.696 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:32.697966 I | op-mon: creating mon b 2022-03-20T15:09:32.836 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:32.837653 I | op-mon: mon "a" ip is 10.107.211.141 2022-03-20T15:09:32.866 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:32.868472 I | op-mon: mon "b" ip is 10.97.13.112 2022-03-20T15:09:32.912 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:32.913984 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.211.141:6789","10.97.13.112:6789"]}] data:a=10.107.211.141:6789,b=10.97.13.112:6789 mapping:{"node":{"a":{"Name":"smithi045","Hostname":"smithi045","Address":"172.21.15.45"},"b":{"Name":"smithi123","Hostname":"smithi123","Address":"172.21.15.123"},"c":{"Name":"smithi006","Hostname":"smithi006","Address":"172.21.15.6"}}} maxMonId:0] 2022-03-20T15:09:32.986 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:32.988410 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T15:09:33.054 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:33.055362 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T15:09:33.320 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:33.322592 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-20T15:09:33.337 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:33.339088 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-20T15:09:33.337 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:33.339122 I | op-mon: waiting for mon quorum with [a b] 2022-03-20T15:09:33.704 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:33.705881 I | op-mon: mons running: [a b] 2022-03-20T15:09:33.928 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:33.930476 I | op-mon: Monitors in quorum: [a] 2022-03-20T15:09:33.965 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:33.967056 I | op-mon: updating maxMonID from 0 to 1 after committing mon "b" 2022-03-20T15:09:34.701 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:34.703117 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.211.141:6789","10.97.13.112:6789"]}] data:a=10.107.211.141:6789,b=10.97.13.112:6789 mapping:{"node":{"a":{"Name":"smithi045","Hostname":"smithi045","Address":"172.21.15.45"},"b":{"Name":"smithi123","Hostname":"smithi123","Address":"172.21.15.123"},"c":{"Name":"smithi006","Hostname":"smithi006","Address":"172.21.15.6"}}} maxMonId:1] 2022-03-20T15:09:34.702 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:34.703159 I | op-mon: waiting for mon quorum with [a b] 2022-03-20T15:09:35.107 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:35.109301 I | op-mon: mons running: [a b] 2022-03-20T15:09:35.331 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:35.333328 I | op-mon: Monitors in quorum: [a] 2022-03-20T15:09:35.331 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:35.333357 I | op-mon: mons created: 2 2022-03-20T15:09:35.401 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T15:09:35.467 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:09:35.468 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-tools-d6d7c985c-54sh5 0/1 ContainerCreating 0 2m32s 2022-03-20T15:09:35.569 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:35.552801 I | op-mon: waiting for mon quorum with [a b] 2022-03-20T15:09:35.569 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:35.568003 I | op-mon: mons running: [a b] 2022-03-20T15:09:35.777 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:35.779139 W | op-mon: failed to find initial monitor b in mon map 2022-03-20T15:09:35.778 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:35.779156 W | op-mon: monitor b is not in quorum list 2022-03-20T15:09:40.470 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T15:09:41.128 INFO:teuthology.orchestra.run.smithi006.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T15:09:41.128 INFO:teuthology.orchestra.run.smithi006.stdout:rook-ceph-tools-d6d7c985c-54sh5 1/1 Running 0 2m38s 2022-03-20T15:09:41.130 INFO:tasks.rook:Waiting for mgr/rook orchestrator to be available 2022-03-20T15:09:41.130 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch status -f json 2022-03-20T15:09:41.624 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:41.622017 I | op-mon: mons running: [a b] 2022-03-20T15:09:41.828 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:41.830169 W | op-mon: failed to find initial monitor b in mon map 2022-03-20T15:09:41.828 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:41.830186 W | op-mon: monitor b is not in quorum list 2022-03-20T15:09:42.573 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:09:42.574 INFO:teuthology.orchestra.run.smithi006.stderr:no valid command found; 10 closest matches: 2022-03-20T15:09:42.574 INFO:teuthology.orchestra.run.smithi006.stderr:pg stat 2022-03-20T15:09:42.575 INFO:teuthology.orchestra.run.smithi006.stderr:pg getmap 2022-03-20T15:09:42.575 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-20T15:09:42.575 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-20T15:09:42.575 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_pools_json 2022-03-20T15:09:42.576 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-pool [...] 2022-03-20T15:09:42.576 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-primary [] [...] 2022-03-20T15:09:42.576 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-osd [] [...] 2022-03-20T15:09:42.576 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls [] [...] 2022-03-20T15:09:42.577 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-20T15:09:42.577 INFO:teuthology.orchestra.run.smithi006.stderr:Error EINVAL: invalid command 2022-03-20T15:09:42.578 INFO:teuthology.orchestra.run.smithi006.stderr:command terminated with exit code 22 2022-03-20T15:09:42.580 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-20T15:09:47.592 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:47.594171 I | op-mon: mon b is not yet running 2022-03-20T15:09:47.592 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:47.594204 I | op-mon: mons running: [a] 2022-03-20T15:09:52.581 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch status -f json 2022-03-20T15:09:52.605 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:52.607170 I | op-mon: mon b is not yet running 2022-03-20T15:09:52.605 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:52.607199 I | op-mon: mons running: [a] 2022-03-20T15:09:53.120 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:09:53.120 INFO:teuthology.orchestra.run.smithi006.stderr:no valid command found; 10 closest matches: 2022-03-20T15:09:53.121 INFO:teuthology.orchestra.run.smithi006.stderr:pg stat 2022-03-20T15:09:53.121 INFO:teuthology.orchestra.run.smithi006.stderr:pg getmap 2022-03-20T15:09:53.121 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-20T15:09:53.121 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-20T15:09:53.122 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_pools_json 2022-03-20T15:09:53.122 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-pool [...] 2022-03-20T15:09:53.122 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-primary [] [...] 2022-03-20T15:09:53.122 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-osd [] [...] 2022-03-20T15:09:53.123 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls [] [...] 2022-03-20T15:09:53.123 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-20T15:09:53.123 INFO:teuthology.orchestra.run.smithi006.stderr:Error EINVAL: invalid command 2022-03-20T15:09:53.124 INFO:teuthology.orchestra.run.smithi006.stderr:command terminated with exit code 22 2022-03-20T15:09:53.127 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-20T15:09:57.996 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:57.998403 I | op-mon: mon b is not yet running 2022-03-20T15:09:57.996 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:09:57.998435 I | op-mon: mons running: [a] 2022-03-20T15:10:03.128 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch status -f json 2022-03-20T15:10:03.466 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:03.468635 I | op-mon: mon b is not yet running 2022-03-20T15:10:03.466 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:03.468663 I | op-mon: mons running: [a] 2022-03-20T15:10:05.110 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:10:05.111 INFO:teuthology.orchestra.run.smithi006.stderr:no valid command found; 10 closest matches: 2022-03-20T15:10:05.111 INFO:teuthology.orchestra.run.smithi006.stderr:pg stat 2022-03-20T15:10:05.111 INFO:teuthology.orchestra.run.smithi006.stderr:pg getmap 2022-03-20T15:10:05.112 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-20T15:10:05.112 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-20T15:10:05.112 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_pools_json 2022-03-20T15:10:05.112 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-pool [...] 2022-03-20T15:10:05.112 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-primary [] [...] 2022-03-20T15:10:05.113 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-osd [] [...] 2022-03-20T15:10:05.113 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls [] [...] 2022-03-20T15:10:05.113 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-20T15:10:05.113 INFO:teuthology.orchestra.run.smithi006.stderr:Error EINVAL: invalid command 2022-03-20T15:10:05.114 INFO:teuthology.orchestra.run.smithi006.stderr:command terminated with exit code 22 2022-03-20T15:10:05.116 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-20T15:10:09.642 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:09.644438 I | op-mon: mon b is not yet running 2022-03-20T15:10:09.642 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:09.644475 I | op-mon: mons running: [a] 2022-03-20T15:10:15.118 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch status -f json 2022-03-20T15:10:15.461 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:15.464140 I | op-mon: mon b is not yet running 2022-03-20T15:10:15.461 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:15.464181 I | op-mon: mons running: [a] 2022-03-20T15:10:16.323 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:10:16.324 INFO:teuthology.orchestra.run.smithi006.stderr:no valid command found; 10 closest matches: 2022-03-20T15:10:16.324 INFO:teuthology.orchestra.run.smithi006.stderr:pg stat 2022-03-20T15:10:16.324 INFO:teuthology.orchestra.run.smithi006.stderr:pg getmap 2022-03-20T15:10:16.325 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-20T15:10:16.325 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-20T15:10:16.325 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_pools_json 2022-03-20T15:10:16.325 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-pool [...] 2022-03-20T15:10:16.326 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-primary [] [...] 2022-03-20T15:10:16.326 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-osd [] [...] 2022-03-20T15:10:16.326 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls [] [...] 2022-03-20T15:10:16.326 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-20T15:10:16.327 INFO:teuthology.orchestra.run.smithi006.stderr:Error EINVAL: invalid command 2022-03-20T15:10:16.329 INFO:teuthology.orchestra.run.smithi006.stderr:command terminated with exit code 22 2022-03-20T15:10:16.331 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-20T15:10:20.518 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:20.517342 I | op-mon: mon b is not yet running 2022-03-20T15:10:20.519 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:20.517377 I | op-mon: mons running: [a] 2022-03-20T15:10:25.656 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:25.659153 I | op-mon: mons running: [a b] 2022-03-20T15:10:26.332 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch status -f json 2022-03-20T15:10:27.724 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:10:27.724 INFO:teuthology.orchestra.run.smithi006.stderr:no valid command found; 10 closest matches: 2022-03-20T15:10:27.725 INFO:teuthology.orchestra.run.smithi006.stderr:pg stat 2022-03-20T15:10:27.725 INFO:teuthology.orchestra.run.smithi006.stderr:pg getmap 2022-03-20T15:10:27.725 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-20T15:10:27.726 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-20T15:10:27.726 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_pools_json 2022-03-20T15:10:27.726 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-pool [...] 2022-03-20T15:10:27.727 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-primary [] [...] 2022-03-20T15:10:27.727 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-osd [] [...] 2022-03-20T15:10:27.727 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls [] [...] 2022-03-20T15:10:27.727 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-20T15:10:27.727 INFO:teuthology.orchestra.run.smithi006.stderr:Error EINVAL: invalid command 2022-03-20T15:10:27.728 INFO:teuthology.orchestra.run.smithi006.stderr:command terminated with exit code 22 2022-03-20T15:10:27.730 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-20T15:10:28.916 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:28.919662 I | op-mon: Monitors in quorum: [a b] 2022-03-20T15:10:28.917 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:28.919685 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T15:10:29.131 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:29.133864 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T15:10:29.131 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:29.133896 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T15:10:29.384 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:29.387110 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T15:10:29.384 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:29.387142 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T15:10:29.623 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:29.626482 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T15:10:29.624 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:29.626508 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-20T15:10:29.863 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:29.866615 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-20T15:10:29.864 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:29.866635 I | op-config: deleting "log file" option from the mon configuration database 2022-03-20T15:10:30.084 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:30.087505 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-20T15:10:30.085 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:30.087534 I | op-mon: creating mon c 2022-03-20T15:10:30.176 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:30.179034 I | op-mon: mon "a" ip is 10.107.211.141 2022-03-20T15:10:30.294 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:30.297464 I | op-mon: mon "b" ip is 10.97.13.112 2022-03-20T15:10:30.323 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:30.326411 I | op-mon: mon "c" ip is 10.99.78.235 2022-03-20T15:10:30.491 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:30.494385 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.211.141:6789","10.97.13.112:6789","10.99.78.235:6789"]}] data:a=10.107.211.141:6789,b=10.97.13.112:6789,c=10.99.78.235:6789 mapping:{"node":{"a":{"Name":"smithi045","Hostname":"smithi045","Address":"172.21.15.45"},"b":{"Name":"smithi123","Hostname":"smithi123","Address":"172.21.15.123"},"c":{"Name":"smithi006","Hostname":"smithi006","Address":"172.21.15.6"}}} maxMonId:1] 2022-03-20T15:10:31.090 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:31.093176 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T15:10:31.090 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:31.093459 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T15:10:31.515 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:31.517077 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-20T15:10:31.525 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:31.527837 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-20T15:10:31.525 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:31.527865 I | op-mon: waiting for mon quorum with [a b c] 2022-03-20T15:10:32.111 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:32.114495 I | op-mon: mons running: [a b c] 2022-03-20T15:10:32.368 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:32.371673 I | op-mon: Monitors in quorum: [a b] 2022-03-20T15:10:32.376 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:32.379032 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-20T15:10:32.385 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:32.388675 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-20T15:10:32.385 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:32.388698 I | op-mon: waiting for mon quorum with [a b c] 2022-03-20T15:10:32.693 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:32.696815 I | op-mon: mons running: [a b c] 2022-03-20T15:10:32.917 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:32.920811 I | op-mon: Monitors in quorum: [a b] 2022-03-20T15:10:32.956 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:32.959685 I | op-mon: updating maxMonID from 1 to 2 after committing mon "c" 2022-03-20T15:10:33.742 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:33.745279 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.13.112:6789","10.99.78.235:6789","10.107.211.141:6789"]}] data:a=10.107.211.141:6789,b=10.97.13.112:6789,c=10.99.78.235:6789 mapping:{"node":{"a":{"Name":"smithi045","Hostname":"smithi045","Address":"172.21.15.45"},"b":{"Name":"smithi123","Hostname":"smithi123","Address":"172.21.15.123"},"c":{"Name":"smithi006","Hostname":"smithi006","Address":"172.21.15.6"}}} maxMonId:2] 2022-03-20T15:10:33.742 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:33.745319 I | op-mon: waiting for mon quorum with [a b c] 2022-03-20T15:10:34.293 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:34.296300 I | op-mon: mons running: [a b c] 2022-03-20T15:10:34.678 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:34.681572 I | op-mon: Monitors in quorum: [a b] 2022-03-20T15:10:34.679 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:34.681599 I | op-mon: mons created: 3 2022-03-20T15:10:34.898 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:34.900854 I | op-mon: waiting for mon quorum with [a b c] 2022-03-20T15:10:34.931 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:34.930661 I | op-mon: mons running: [a b c] 2022-03-20T15:10:35.156 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:35.159409 W | op-mon: failed to find initial monitor c in mon map 2022-03-20T15:10:35.156 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:35.159438 W | op-mon: monitor c is not in quorum list 2022-03-20T15:10:37.732 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch status -f json 2022-03-20T15:10:38.226 INFO:teuthology.orchestra.run.smithi006.stderr:no valid command found; 10 closest matches: 2022-03-20T15:10:38.226 INFO:teuthology.orchestra.run.smithi006.stderr:pg stat 2022-03-20T15:10:38.227 INFO:teuthology.orchestra.run.smithi006.stderr:pg getmap 2022-03-20T15:10:38.227 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-20T15:10:38.227 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-20T15:10:38.227 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_pools_json 2022-03-20T15:10:38.228 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-pool [...] 2022-03-20T15:10:38.228 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-primary [] [...] 2022-03-20T15:10:38.228 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-osd [] [...] 2022-03-20T15:10:38.228 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls [] [...] 2022-03-20T15:10:38.228 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-20T15:10:38.229 INFO:teuthology.orchestra.run.smithi006.stderr:Error EINVAL: invalid command 2022-03-20T15:10:38.229 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:10:38.230 INFO:teuthology.orchestra.run.smithi006.stderr:command terminated with exit code 22 2022-03-20T15:10:38.232 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-20T15:10:40.179 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:40.182314 I | op-mon: mons running: [a b c] 2022-03-20T15:10:40.412 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:40.415119 W | op-mon: failed to find initial monitor c in mon map 2022-03-20T15:10:40.412 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:40.415151 W | op-mon: monitor c is not in quorum list 2022-03-20T15:10:45.442 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:45.441571 I | op-mon: mons running: [a b c] 2022-03-20T15:10:45.678 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:45.681373 W | op-mon: failed to find initial monitor c in mon map 2022-03-20T15:10:45.678 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:45.681406 W | op-mon: monitor c is not in quorum list 2022-03-20T15:10:48.234 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch status -f json 2022-03-20T15:10:48.740 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:10:48.741 INFO:teuthology.orchestra.run.smithi006.stderr:no valid command found; 10 closest matches: 2022-03-20T15:10:48.741 INFO:teuthology.orchestra.run.smithi006.stderr:pg stat 2022-03-20T15:10:48.741 INFO:teuthology.orchestra.run.smithi006.stderr:pg getmap 2022-03-20T15:10:48.741 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-20T15:10:48.742 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-20T15:10:48.742 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_pools_json 2022-03-20T15:10:48.742 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-pool [...] 2022-03-20T15:10:48.742 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-primary [] [...] 2022-03-20T15:10:48.743 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-osd [] [...] 2022-03-20T15:10:48.743 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls [] [...] 2022-03-20T15:10:48.743 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-20T15:10:48.743 INFO:teuthology.orchestra.run.smithi006.stderr:Error EINVAL: invalid command 2022-03-20T15:10:48.745 INFO:teuthology.orchestra.run.smithi006.stderr:command terminated with exit code 22 2022-03-20T15:10:48.748 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-20T15:10:50.761 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:50.764763 I | op-mon: mons running: [a b c] 2022-03-20T15:10:50.987 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:50.990544 W | op-mon: failed to find initial monitor c in mon map 2022-03-20T15:10:50.988 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:50.990575 W | op-mon: monitor c is not in quorum list 2022-03-20T15:10:56.008 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:56.011436 I | op-mon: mon c is not yet running 2022-03-20T15:10:56.008 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:10:56.011460 I | op-mon: mons running: [a b] 2022-03-20T15:10:58.749 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch status -f json 2022-03-20T15:10:59.267 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:10:59.268 INFO:teuthology.orchestra.run.smithi006.stderr:no valid command found; 10 closest matches: 2022-03-20T15:10:59.268 INFO:teuthology.orchestra.run.smithi006.stderr:pg stat 2022-03-20T15:10:59.268 INFO:teuthology.orchestra.run.smithi006.stderr:pg getmap 2022-03-20T15:10:59.268 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-20T15:10:59.269 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-20T15:10:59.269 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_pools_json 2022-03-20T15:10:59.269 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-pool [...] 2022-03-20T15:10:59.269 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-primary [] [...] 2022-03-20T15:10:59.270 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-osd [] [...] 2022-03-20T15:10:59.270 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls [] [...] 2022-03-20T15:10:59.270 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-20T15:10:59.270 INFO:teuthology.orchestra.run.smithi006.stderr:Error EINVAL: invalid command 2022-03-20T15:10:59.271 INFO:teuthology.orchestra.run.smithi006.stderr:command terminated with exit code 22 2022-03-20T15:10:59.273 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-20T15:11:01.031 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:01.034660 I | op-mon: mon c is not yet running 2022-03-20T15:11:01.031 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:01.034690 I | op-mon: mons running: [a b] 2022-03-20T15:11:06.052 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:06.056554 I | op-mon: mon c is not yet running 2022-03-20T15:11:06.053 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:06.056580 I | op-mon: mons running: [a b] 2022-03-20T15:11:09.274 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch status -f json 2022-03-20T15:11:09.757 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:11:09.758 INFO:teuthology.orchestra.run.smithi006.stderr:no valid command found; 10 closest matches: 2022-03-20T15:11:09.758 INFO:teuthology.orchestra.run.smithi006.stderr:pg stat 2022-03-20T15:11:09.759 INFO:teuthology.orchestra.run.smithi006.stderr:pg getmap 2022-03-20T15:11:09.759 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-20T15:11:09.759 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-20T15:11:09.759 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_pools_json 2022-03-20T15:11:09.760 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-pool [...] 2022-03-20T15:11:09.760 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-primary [] [...] 2022-03-20T15:11:09.760 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-osd [] [...] 2022-03-20T15:11:09.760 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls [] [...] 2022-03-20T15:11:09.761 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-20T15:11:09.761 INFO:teuthology.orchestra.run.smithi006.stderr:Error EINVAL: invalid command 2022-03-20T15:11:09.762 INFO:teuthology.orchestra.run.smithi006.stderr:command terminated with exit code 22 2022-03-20T15:11:09.765 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-20T15:11:11.083 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:11.082952 I | op-mon: mon c is not yet running 2022-03-20T15:11:11.083 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:11.082988 I | op-mon: mons running: [a b] 2022-03-20T15:11:16.101 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:16.105145 I | op-mon: mon c is not yet running 2022-03-20T15:11:16.102 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:16.105185 I | op-mon: mons running: [a b] 2022-03-20T15:11:19.767 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch status -f json 2022-03-20T15:11:20.247 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:11:20.248 INFO:teuthology.orchestra.run.smithi006.stderr:no valid command found; 10 closest matches: 2022-03-20T15:11:20.248 INFO:teuthology.orchestra.run.smithi006.stderr:pg stat 2022-03-20T15:11:20.248 INFO:teuthology.orchestra.run.smithi006.stderr:pg getmap 2022-03-20T15:11:20.248 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-20T15:11:20.248 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-20T15:11:20.249 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_pools_json 2022-03-20T15:11:20.249 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-pool [...] 2022-03-20T15:11:20.249 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-primary [] [...] 2022-03-20T15:11:20.249 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-osd [] [...] 2022-03-20T15:11:20.250 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls [] [...] 2022-03-20T15:11:20.250 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-20T15:11:20.250 INFO:teuthology.orchestra.run.smithi006.stderr:Error EINVAL: invalid command 2022-03-20T15:11:20.251 INFO:teuthology.orchestra.run.smithi006.stderr:command terminated with exit code 22 2022-03-20T15:11:20.253 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-20T15:11:21.131 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:21.132169 I | op-mon: mon c is not yet running 2022-03-20T15:11:21.131 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:21.132204 I | op-mon: mons running: [a b] 2022-03-20T15:11:26.151 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:26.155320 I | op-mon: mon c is not yet running 2022-03-20T15:11:26.151 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:26.155355 I | op-mon: mons running: [a b] 2022-03-20T15:11:30.255 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch status -f json 2022-03-20T15:11:30.730 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:11:30.730 INFO:teuthology.orchestra.run.smithi006.stderr:no valid command found; 10 closest matches: 2022-03-20T15:11:30.731 INFO:teuthology.orchestra.run.smithi006.stderr:pg stat 2022-03-20T15:11:30.731 INFO:teuthology.orchestra.run.smithi006.stderr:pg getmap 2022-03-20T15:11:30.731 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-20T15:11:30.731 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-20T15:11:30.731 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_pools_json 2022-03-20T15:11:30.732 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-pool [...] 2022-03-20T15:11:30.732 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-primary [] [...] 2022-03-20T15:11:30.732 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-osd [] [...] 2022-03-20T15:11:30.732 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls [] [...] 2022-03-20T15:11:30.733 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-20T15:11:30.733 INFO:teuthology.orchestra.run.smithi006.stderr:Error EINVAL: invalid command 2022-03-20T15:11:30.734 INFO:teuthology.orchestra.run.smithi006.stderr:command terminated with exit code 22 2022-03-20T15:11:30.735 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-20T15:11:31.189 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:31.193694 I | op-mon: mon c is not yet running 2022-03-20T15:11:31.189 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:31.193728 I | op-mon: mons running: [a b] 2022-03-20T15:11:37.248 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:37.252971 I | op-mon: mon c is not yet running 2022-03-20T15:11:37.248 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:37.253009 I | op-mon: mons running: [a b] 2022-03-20T15:11:40.738 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch status -f json 2022-03-20T15:11:41.271 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:11:41.271 INFO:teuthology.orchestra.run.smithi006.stderr:no valid command found; 10 closest matches: 2022-03-20T15:11:41.272 INFO:teuthology.orchestra.run.smithi006.stderr:pg stat 2022-03-20T15:11:41.272 INFO:teuthology.orchestra.run.smithi006.stderr:pg getmap 2022-03-20T15:11:41.272 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-20T15:11:41.272 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-20T15:11:41.273 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_pools_json 2022-03-20T15:11:41.273 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-pool [...] 2022-03-20T15:11:41.273 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-primary [] [...] 2022-03-20T15:11:41.273 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-osd [] [...] 2022-03-20T15:11:41.273 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls [] [...] 2022-03-20T15:11:41.274 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-20T15:11:41.274 INFO:teuthology.orchestra.run.smithi006.stderr:Error EINVAL: invalid command 2022-03-20T15:11:41.274 INFO:teuthology.orchestra.run.smithi006.stderr:command terminated with exit code 22 2022-03-20T15:11:41.276 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-20T15:11:43.018 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:43.022649 I | op-mon: mon c is not yet running 2022-03-20T15:11:43.018 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:43.022687 I | op-mon: mons running: [a b] 2022-03-20T15:11:48.074 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:48.079341 I | op-mon: mon c is not yet running 2022-03-20T15:11:48.075 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:48.079376 I | op-mon: mons running: [a b] 2022-03-20T15:11:51.281 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch status -f json 2022-03-20T15:11:53.126 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:53.131551 I | op-mon: mons running: [a b c] 2022-03-20T15:11:56.317 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:11:56.318 INFO:teuthology.orchestra.run.smithi006.stderr:no valid command found; 10 closest matches: 2022-03-20T15:11:56.318 INFO:teuthology.orchestra.run.smithi006.stderr:pg stat 2022-03-20T15:11:56.318 INFO:teuthology.orchestra.run.smithi006.stderr:pg getmap 2022-03-20T15:11:56.318 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-20T15:11:56.319 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-20T15:11:56.319 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_pools_json 2022-03-20T15:11:56.319 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-pool [...] 2022-03-20T15:11:56.319 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-primary [] [...] 2022-03-20T15:11:56.320 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-osd [] [...] 2022-03-20T15:11:56.320 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls [] [...] 2022-03-20T15:11:56.320 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-20T15:11:56.320 INFO:teuthology.orchestra.run.smithi006.stderr:Error EINVAL: invalid command 2022-03-20T15:11:56.321 INFO:teuthology.orchestra.run.smithi006.stderr:command terminated with exit code 22 2022-03-20T15:11:56.324 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-20T15:11:56.427 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:56.432460 I | op-mon: Monitors in quorum: [a b c] 2022-03-20T15:11:56.428 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:56.432493 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T15:11:56.653 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:56.655398 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T15:11:56.653 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:56.655431 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T15:11:56.869 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:56.874806 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T15:11:56.870 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:56.874845 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T15:11:57.086 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:57.091341 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T15:11:57.086 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:57.091379 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-20T15:11:57.291 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:57.296171 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-20T15:11:57.291 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:57.296195 I | op-config: deleting "log file" option from the mon configuration database 2022-03-20T15:11:57.569 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:57.574351 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-20T15:11:57.570 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:57.574401 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-20T15:11:57.929 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:57.934468 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-20T15:11:58.372 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:58.377357 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-20T15:11:58.964 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:58.968780 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-20T15:11:59.429 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:59.432784 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-20T15:11:59.430 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:59.432815 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-20T15:11:59.881 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:59.886266 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-20T15:11:59.882 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:59.886303 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-20T15:11:59.882 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:11:59.886313 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-20T15:12:00.299 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:00.304727 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-20T15:12:00.321 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:00.326157 I | op-mgr: start running mgr 2022-03-20T15:12:00.321 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:00.326198 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-20T15:12:05.310 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:05.315173 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi123": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:12:06.326 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch status -f json 2022-03-20T15:12:06.779 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:12:06.780 INFO:teuthology.orchestra.run.smithi006.stderr:no valid command found; 10 closest matches: 2022-03-20T15:12:06.780 INFO:teuthology.orchestra.run.smithi006.stderr:pg stat 2022-03-20T15:12:06.780 INFO:teuthology.orchestra.run.smithi006.stderr:pg getmap 2022-03-20T15:12:06.780 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-20T15:12:06.781 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-20T15:12:06.781 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_pools_json 2022-03-20T15:12:06.781 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-pool [...] 2022-03-20T15:12:06.782 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-primary [] [...] 2022-03-20T15:12:06.782 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-osd [] [...] 2022-03-20T15:12:06.782 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls [] [...] 2022-03-20T15:12:06.782 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-20T15:12:06.783 INFO:teuthology.orchestra.run.smithi006.stderr:Error EINVAL: invalid command 2022-03-20T15:12:06.783 INFO:teuthology.orchestra.run.smithi006.stderr:command terminated with exit code 22 2022-03-20T15:12:06.785 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-20T15:12:16.787 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch status -f json 2022-03-20T15:12:17.282 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:12:17.282 INFO:teuthology.orchestra.run.smithi006.stderr:no valid command found; 10 closest matches: 2022-03-20T15:12:17.283 INFO:teuthology.orchestra.run.smithi006.stderr:pg stat 2022-03-20T15:12:17.283 INFO:teuthology.orchestra.run.smithi006.stderr:pg getmap 2022-03-20T15:12:17.283 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-20T15:12:17.283 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-20T15:12:17.284 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_pools_json 2022-03-20T15:12:17.284 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-pool [...] 2022-03-20T15:12:17.284 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-primary [] [...] 2022-03-20T15:12:17.284 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls-by-osd [] [...] 2022-03-20T15:12:17.285 INFO:teuthology.orchestra.run.smithi006.stderr:pg ls [] [...] 2022-03-20T15:12:17.285 INFO:teuthology.orchestra.run.smithi006.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-20T15:12:17.285 INFO:teuthology.orchestra.run.smithi006.stderr:Error EINVAL: invalid command 2022-03-20T15:12:17.286 INFO:teuthology.orchestra.run.smithi006.stderr:command terminated with exit code 22 2022-03-20T15:12:17.288 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-20T15:12:22.028 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:22.034218 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mgr-a" 2022-03-20T15:12:22.134 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:22.038595 I | op-mgr: setting services to point to mgr "a" 2022-03-20T15:12:22.134 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:22.103357 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-20T15:12:22.135 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:22.103471 I | op-mgr: successful modules: balancer 2022-03-20T15:12:22.135 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:22.111580 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-20T15:12:22.135 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:22.111613 W | op-osd: useAllNodes is set to false and no nodes, storageClassDevicesets or volumeSources are specified, no OSD pods are going to be created 2022-03-20T15:12:22.136 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:22.111627 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-20T15:12:22.189 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:22.195146 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-20T15:12:22.193 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:22.198853 I | op-osd: no storageClassDeviceSets defined to configure OSDs on PVCs 2022-03-20T15:12:22.193 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:22.198886 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-20T15:12:22.193 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:22.198892 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-20T15:12:22.674 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:22.679938 I | op-mgr: successful modules: dashboard 2022-03-20T15:12:22.676 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:22.681075 I | op-mgr: successful modules: prometheus 2022-03-20T15:12:22.767 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:22.772875 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-20T15:12:22.767 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:22.772900 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-20T15:12:22.775 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:22.781448 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph" 2022-03-20T15:12:22.776 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:22.781484 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "rook-ceph" 2022-03-20T15:12:22.776 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:22.781518 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph" 2022-03-20T15:12:24.024 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:24.030187 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-20T15:12:27.290 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch status -f json 2022-03-20T15:12:38.916 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:38.920726 I | op-mgr: successful modules: orchestrator modules 2022-03-20T15:12:38.978 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:12:38.978 INFO:teuthology.orchestra.run.smithi006.stdout:{"available": true, "backend": "rook"} 2022-03-20T15:12:39.007 INFO:tasks.rook: mgr/rook orchestrator is active 2022-03-20T15:12:39.008 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph config set mgr mgr/rook/storage_class scratch 2022-03-20T15:12:39.652 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch apply osd --all-available-devices 2022-03-20T15:12:40.415 INFO:teuthology.orchestra.run.smithi006.stdout:['Success'] 2022-03-20T15:12:40.416 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.419446 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-20T15:12:40.416 INFO:tasks.rook.operator.smithi006.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", AllowUnsupported: true}, 2022-03-20T15:12:40.417 INFO:tasks.rook.operator.smithi006.stdout: Storage: v1.StorageScopeSpec{ 2022-03-20T15:12:40.417 INFO:tasks.rook.operator.smithi006.stdout: ... // 3 identical fields 2022-03-20T15:12:40.417 INFO:tasks.rook.operator.smithi006.stdout: Config: nil, 2022-03-20T15:12:40.417 INFO:tasks.rook.operator.smithi006.stdout: Selection: {}, 2022-03-20T15:12:40.418 INFO:tasks.rook.operator.smithi006.stdout:- StorageClassDeviceSets: nil, 2022-03-20T15:12:40.418 INFO:tasks.rook.operator.smithi006.stdout:+ StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-20T15:12:40.418 INFO:tasks.rook.operator.smithi006.stdout:+ { 2022-03-20T15:12:40.424 INFO:tasks.rook.operator.smithi006.stdout:+ Name: "smithi006-lv-1", 2022-03-20T15:12:40.424 INFO:tasks.rook.operator.smithi006.stdout:+ Count: 1, 2022-03-20T15:12:40.424 INFO:tasks.rook.operator.smithi006.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-20T15:12:40.425 INFO:tasks.rook.operator.smithi006.stdout:+ }, 2022-03-20T15:12:40.425 INFO:tasks.rook.operator.smithi006.stdout:+ { 2022-03-20T15:12:40.425 INFO:tasks.rook.operator.smithi006.stdout:+ Name: "smithi006-lv-2", 2022-03-20T15:12:40.426 INFO:tasks.rook.operator.smithi006.stdout:+ Count: 1, 2022-03-20T15:12:40.426 INFO:tasks.rook.operator.smithi006.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-20T15:12:40.426 INFO:tasks.rook.operator.smithi006.stdout:+ }, 2022-03-20T15:12:40.427 INFO:tasks.rook.operator.smithi006.stdout:+ { 2022-03-20T15:12:40.427 INFO:tasks.rook.operator.smithi006.stdout:+ Name: "smithi006-lv-3", 2022-03-20T15:12:40.427 INFO:tasks.rook.operator.smithi006.stdout:+ Count: 1, 2022-03-20T15:12:40.427 INFO:tasks.rook.operator.smithi006.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-20T15:12:40.428 INFO:tasks.rook.operator.smithi006.stdout:+ }, 2022-03-20T15:12:40.428 INFO:tasks.rook.operator.smithi006.stdout:+ { 2022-03-20T15:12:40.428 INFO:tasks.rook.operator.smithi006.stdout:+ Name: "smithi006-lv-4", 2022-03-20T15:12:40.428 INFO:tasks.rook.operator.smithi006.stdout:+ Count: 1, 2022-03-20T15:12:40.429 INFO:tasks.rook.operator.smithi006.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-20T15:12:40.429 INFO:tasks.rook.operator.smithi006.stdout:+ }, 2022-03-20T15:12:40.429 INFO:tasks.rook.operator.smithi006.stdout:+ { 2022-03-20T15:12:40.429 INFO:tasks.rook.operator.smithi006.stdout:+ Name: "smithi045-lv-1", 2022-03-20T15:12:40.429 INFO:tasks.rook.operator.smithi006.stdout:+ Count: 1, 2022-03-20T15:12:40.430 INFO:tasks.rook.operator.smithi006.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-20T15:12:40.430 INFO:tasks.rook.operator.smithi006.stdout:+ }, 2022-03-20T15:12:40.430 INFO:tasks.rook.operator.smithi006.stdout:+ { 2022-03-20T15:12:40.430 INFO:tasks.rook.operator.smithi006.stdout:+ Name: "smithi045-lv-2", 2022-03-20T15:12:40.431 INFO:tasks.rook.operator.smithi006.stdout:+ Count: 1, 2022-03-20T15:12:40.431 INFO:tasks.rook.operator.smithi006.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-20T15:12:40.431 INFO:tasks.rook.operator.smithi006.stdout:+ }, 2022-03-20T15:12:40.431 INFO:tasks.rook.operator.smithi006.stdout:+ { 2022-03-20T15:12:40.431 INFO:tasks.rook.operator.smithi006.stdout:+ Name: "smithi045-lv-3", 2022-03-20T15:12:40.432 INFO:tasks.rook.operator.smithi006.stdout:+ Count: 1, 2022-03-20T15:12:40.432 INFO:tasks.rook.operator.smithi006.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-20T15:12:40.432 INFO:tasks.rook.operator.smithi006.stdout:+ }, 2022-03-20T15:12:40.432 INFO:tasks.rook.operator.smithi006.stdout:+ { 2022-03-20T15:12:40.433 INFO:tasks.rook.operator.smithi006.stdout:+ Name: "smithi045-lv-4", 2022-03-20T15:12:40.433 INFO:tasks.rook.operator.smithi006.stdout:+ Count: 1, 2022-03-20T15:12:40.433 INFO:tasks.rook.operator.smithi006.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-20T15:12:40.433 INFO:tasks.rook.operator.smithi006.stdout:+ }, 2022-03-20T15:12:40.433 INFO:tasks.rook.operator.smithi006.stdout:+ { 2022-03-20T15:12:40.434 INFO:tasks.rook.operator.smithi006.stdout:+ Name: "smithi123-lv-1", 2022-03-20T15:12:40.434 INFO:tasks.rook.operator.smithi006.stdout:+ Count: 1, 2022-03-20T15:12:40.434 INFO:tasks.rook.operator.smithi006.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-20T15:12:40.434 INFO:tasks.rook.operator.smithi006.stdout:+ }, 2022-03-20T15:12:40.435 INFO:tasks.rook.operator.smithi006.stdout:+ { 2022-03-20T15:12:40.435 INFO:tasks.rook.operator.smithi006.stdout:+ Name: "smithi123-lv-2", 2022-03-20T15:12:40.435 INFO:tasks.rook.operator.smithi006.stdout:+ Count: 1, 2022-03-20T15:12:40.435 INFO:tasks.rook.operator.smithi006.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-20T15:12:40.436 INFO:tasks.rook.operator.smithi006.stdout:+ }, 2022-03-20T15:12:40.436 INFO:tasks.rook.operator.smithi006.stdout:+ { 2022-03-20T15:12:40.436 INFO:tasks.rook.operator.smithi006.stdout:+ Name: "smithi123-lv-3", 2022-03-20T15:12:40.436 INFO:tasks.rook.operator.smithi006.stdout:+ Count: 1, 2022-03-20T15:12:40.437 INFO:tasks.rook.operator.smithi006.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-20T15:12:40.437 INFO:tasks.rook.operator.smithi006.stdout:+ }, 2022-03-20T15:12:40.437 INFO:tasks.rook.operator.smithi006.stdout:+ { 2022-03-20T15:12:40.437 INFO:tasks.rook.operator.smithi006.stdout:+ Name: "smithi123-lv-4", 2022-03-20T15:12:40.437 INFO:tasks.rook.operator.smithi006.stdout:+ Count: 1, 2022-03-20T15:12:40.438 INFO:tasks.rook.operator.smithi006.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-20T15:12:40.438 INFO:tasks.rook.operator.smithi006.stdout:+ }, 2022-03-20T15:12:40.438 INFO:tasks.rook.operator.smithi006.stdout:+ }, 2022-03-20T15:12:40.438 INFO:tasks.rook.operator.smithi006.stdout: }, 2022-03-20T15:12:40.439 INFO:tasks.rook.operator.smithi006.stdout: Annotations: nil, 2022-03-20T15:12:40.439 INFO:tasks.rook.operator.smithi006.stdout: Labels: nil, 2022-03-20T15:12:40.439 INFO:tasks.rook.operator.smithi006.stdout: ... // 20 identical fields 2022-03-20T15:12:40.439 INFO:tasks.rook.operator.smithi006.stdout: } 2022-03-20T15:12:40.440 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.419541 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-20T15:12:40.440 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.419909 I | operator: successfully started the controller-runtime manager 2022-03-20T15:12:40.440 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.419930 I | ceph-cluster-controller: stopping monitoring of ceph status 2022-03-20T15:12:40.440 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.419940 I | op-mon: stopping monitoring of mons in namespace "rook-ceph" 2022-03-20T15:12:40.440 INFO:tasks.rook.operator.smithi006.stdout:I0320 15:12:40.419941 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-20T15:12:40.441 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.419947 I | op-osd: stopping monitoring of OSDs in namespace "rook-ceph" 2022-03-20T15:12:40.442 INFO:tasks.rook:Waiting for 12 OSDs 2022-03-20T15:12:40.442 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd stat -f json 2022-03-20T15:12:40.469 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.473221 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-20T15:12:40.469 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.473244 I | operator: watching all namespaces for Ceph CRs 2022-03-20T15:12:40.470 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.473269 I | operator: setting up schemes 2022-03-20T15:12:40.474 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.475612 I | operator: setting up the controller-runtime manager 2022-03-20T15:12:40.775 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.779247 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-20T15:12:40.780 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.782185 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-20T15:12:40.786 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.782223 I | ceph-cluster-controller: successfully started 2022-03-20T15:12:40.786 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.782279 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-20T15:12:40.786 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.782303 I | ceph-crashcollector-controller: successfully started 2022-03-20T15:12:40.787 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.782337 I | ceph-block-pool-controller: successfully started 2022-03-20T15:12:40.787 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.782392 I | ceph-object-store-user-controller: successfully started 2022-03-20T15:12:40.787 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.782421 I | ceph-object-realm-controller: successfully started 2022-03-20T15:12:40.787 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.782440 I | ceph-object-zonegroup-controller: successfully started 2022-03-20T15:12:40.788 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.782458 I | ceph-object-zone-controller: successfully started 2022-03-20T15:12:40.788 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.782567 I | ceph-object-controller: successfully started 2022-03-20T15:12:40.788 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.782596 I | ceph-file-controller: successfully started 2022-03-20T15:12:40.788 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.782617 I | ceph-nfs-controller: successfully started 2022-03-20T15:12:40.788 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.782640 I | ceph-rbd-mirror-controller: successfully started 2022-03-20T15:12:40.789 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.782657 I | ceph-client-controller: successfully started 2022-03-20T15:12:40.789 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.782671 I | ceph-filesystem-mirror-controller: successfully started 2022-03-20T15:12:40.789 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.782694 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-20T15:12:40.789 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.782709 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-20T15:12:40.790 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.782722 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-20T15:12:40.790 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.782734 I | ceph-bucket-topic: successfully started 2022-03-20T15:12:40.790 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.782744 I | ceph-bucket-notification: successfully started 2022-03-20T15:12:40.790 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.782752 I | ceph-bucket-notification: successfully started 2022-03-20T15:12:40.791 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.782762 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-20T15:12:40.791 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.783854 I | operator: starting the controller-runtime manager 2022-03-20T15:12:40.884 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.886723 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-20T15:12:40.884 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.886780 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-20T15:12:40.884 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.886799 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-20T15:12:40.888 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.887349 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-20T15:12:40.889 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.891719 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-20T15:12:40.889 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.892478 I | op-mon: parsing mon endpoints: a=10.107.211.141:6789,b=10.97.13.112:6789,c=10.99.78.235:6789 2022-03-20T15:12:40.889 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.892522 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-20T15:12:40.889 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.892532 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-20T15:12:40.892 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.893148 I | op-mon: parsing mon endpoints: a=10.107.211.141:6789,b=10.97.13.112:6789,c=10.99.78.235:6789 2022-03-20T15:12:40.892 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.893884 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-20T15:12:40.892 INFO:tasks.rook.operator.smithi006.stdout:I0320 15:12:40.893936 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-20T15:12:40.903 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.907195 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-20T15:12:40.906 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.909065 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e... 2022-03-20T15:12:40.932 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.933876 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-20T15:12:40.932 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.933936 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-20T15:12:40.932 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.933952 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-20T15:12:40.932 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.933965 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-20T15:12:40.933 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.933980 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-20T15:12:40.933 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.933994 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-20T15:12:40.933 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.934011 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-20T15:12:40.933 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.934027 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-20T15:12:40.933 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.934041 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-20T15:12:40.934 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.934056 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-20T15:12:40.934 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.934068 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-20T15:12:40.934 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.934083 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-20T15:12:40.934 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.934098 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-20T15:12:40.935 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.934112 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-20T15:12:40.935 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.934124 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-20T15:12:40.935 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.934139 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-20T15:12:40.935 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.934204 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-20T15:12:40.935 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:40.934217 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-20T15:12:40.971 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:12:40.996 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":3,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-20T15:12:40.996 INFO:tasks.rook: have 0/12 OSDs 2022-03-20T15:12:46.971 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:46.974414 I | ceph-spec: detected ceph image version: "17.0.0-11187 quincy" 2022-03-20T15:12:46.972 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:46.974451 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-20T15:12:47.220 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.222933 I | op-mon: parsing mon endpoints: a=10.107.211.141:6789,b=10.97.13.112:6789,c=10.99.78.235:6789 2022-03-20T15:12:47.224 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.226965 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T15:12:47.225 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.227226 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T15:12:47.654 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.657212 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11187 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e" 2022-03-20T15:12:47.850 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.851340 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-20T15:12:47.850 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.851378 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-20T15:12:47.850 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.851389 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-20T15:12:47.851 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.851396 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-20T15:12:47.851 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.851403 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-20T15:12:47.851 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.851409 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-20T15:12:47.851 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.851415 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-20T15:12:47.851 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.851421 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-20T15:12:47.852 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.851428 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-20T15:12:47.852 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.851434 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-20T15:12:47.852 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.851441 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-20T15:12:47.852 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.851447 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-20T15:12:47.853 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.851453 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-20T15:12:47.853 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.851460 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-20T15:12:47.853 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.851466 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-20T15:12:47.853 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.851473 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-20T15:12:47.854 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.851484 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-20T15:12:47.854 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.851495 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-20T15:12:47.854 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.851505 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-20T15:12:47.854 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.851517 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-20T15:12:47.854 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.851530 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-20T15:12:47.855 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.851541 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-20T15:12:47.855 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.851552 I | ceph-csi: Kubernetes version is 1.23 2022-03-20T15:12:47.855 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.851570 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-20T15:12:47.855 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.851581 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-20T15:12:47.862 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.865786 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-20T15:12:47.867 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.870474 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-20T15:12:47.868 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.870492 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-20T15:12:47.868 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.870495 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-20T15:12:47.868 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.870498 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-20T15:12:47.868 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.870507 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-20T15:12:47.869 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.870509 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-20T15:12:47.869 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.870512 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-20T15:12:47.888 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:47.891724 I | op-mon: start running mons 2022-03-20T15:12:48.030 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:48.032579 I | op-mon: parsing mon endpoints: a=10.107.211.141:6789,b=10.97.13.112:6789,c=10.99.78.235:6789 2022-03-20T15:12:48.044 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:48.046798 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-20T15:12:48.044 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:48.046819 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-20T15:12:48.044 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:48.046823 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-20T15:12:48.171 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:48.173748 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.99.78.235:6789","10.107.211.141:6789","10.97.13.112:6789"]}] data:a=10.107.211.141:6789,b=10.97.13.112:6789,c=10.99.78.235:6789 mapping:{"node":{"a":{"Name":"smithi045","Hostname":"smithi045","Address":"172.21.15.45"},"b":{"Name":"smithi123","Hostname":"smithi123","Address":"172.21.15.123"},"c":{"Name":"smithi006","Hostname":"smithi006","Address":"172.21.15.6"}}} maxMonId:2] 2022-03-20T15:12:48.197 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:48.200447 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-20T15:12:49.311 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:49.313687 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T15:12:49.316 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:49.314057 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T15:12:49.467 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:49.469960 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-20T15:12:49.468 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:49.469993 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-20T15:12:49.468 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:49.470002 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-20T15:12:49.535 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:49.537981 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-20T15:12:49.535 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:49.538027 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-20T15:12:49.536 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:49.538037 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-20T15:12:49.606 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:49.609106 I | ceph-csi: successfully started CSI CephFS driver 2022-03-20T15:12:50.752 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:50.754777 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-20T15:12:50.757 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:50.760263 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-20T15:12:50.758 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:50.760286 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-20T15:12:50.763 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:50.765235 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-20T15:12:50.998 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd stat -f json 2022-03-20T15:12:51.542 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:12:51.551 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:51.553962 I | op-mon: targeting the mon count 3 2022-03-20T15:12:51.555 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:51.558549 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T15:12:51.562 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":3,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-20T15:12:51.562 INFO:tasks.rook: have 0/12 OSDs 2022-03-20T15:12:51.988 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:51.991086 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T15:12:51.989 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:51.991117 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T15:12:52.355 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:52.357766 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T15:12:52.356 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:52.357799 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T15:12:52.695 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:52.697535 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T15:12:52.695 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:52.697567 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-20T15:12:53.033 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:53.035590 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-20T15:12:53.033 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:53.035622 I | op-config: deleting "log file" option from the mon configuration database 2022-03-20T15:12:53.484 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:53.486483 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-20T15:12:53.484 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:53.486516 I | op-mon: checking for basic quorum with existing mons 2022-03-20T15:12:53.614 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:53.617027 I | op-mon: mon "b" ip is 10.97.13.112 2022-03-20T15:12:53.702 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:53.704450 I | op-mon: mon "c" ip is 10.99.78.235 2022-03-20T15:12:53.952 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:53.954314 I | op-mon: mon "a" ip is 10.107.211.141 2022-03-20T15:12:54.552 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:54.554019 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.211.141:6789","10.97.13.112:6789","10.99.78.235:6789"]}] data:c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 mapping:{"node":{"a":{"Name":"smithi045","Hostname":"smithi045","Address":"172.21.15.45"},"b":{"Name":"smithi123","Hostname":"smithi123","Address":"172.21.15.123"},"c":{"Name":"smithi006","Hostname":"smithi006","Address":"172.21.15.6"}}} maxMonId:2] 2022-03-20T15:12:55.151 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:55.153417 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T15:12:55.152 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:55.153677 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T15:12:55.568 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:55.570105 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-20T15:12:55.577 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:55.580391 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-20T15:12:55.578 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:55.580414 I | op-mon: waiting for mon quorum with [b c a] 2022-03-20T15:12:56.157 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:56.159515 I | op-mon: mons running: [b c a] 2022-03-20T15:12:56.578 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:56.580329 I | op-mon: Monitors in quorum: [a b c] 2022-03-20T15:12:56.587 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:56.589386 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-20T15:12:56.596 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:56.598297 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-20T15:12:56.596 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:56.598315 I | op-mon: waiting for mon quorum with [b c a] 2022-03-20T15:12:56.756 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:56.758167 I | op-mon: mons running: [b c a] 2022-03-20T15:12:57.163 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:57.165797 I | op-mon: Monitors in quorum: [a b c] 2022-03-20T15:12:57.172 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:57.174237 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-20T15:12:57.183 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:57.184999 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-20T15:12:57.183 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:57.185028 I | op-mon: waiting for mon quorum with [b c a] 2022-03-20T15:12:57.356 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:57.358024 I | op-mon: mons running: [b c a] 2022-03-20T15:12:57.809 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:57.810939 I | op-mon: Monitors in quorum: [a b c] 2022-03-20T15:12:57.809 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:57.810969 I | op-mon: mons created: 3 2022-03-20T15:12:58.206 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:58.208130 I | op-mon: waiting for mon quorum with [b c a] 2022-03-20T15:12:58.229 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:58.231471 I | op-mon: mons running: [b c a] 2022-03-20T15:12:58.625 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:58.627208 I | op-mon: Monitors in quorum: [a b c] 2022-03-20T15:12:58.626 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:58.627259 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-20T15:12:59.050 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:59.052135 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-20T15:12:59.496 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:59.498405 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-20T15:12:59.905 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:12:59.907247 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-20T15:13:00.370 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:00.372080 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-20T15:13:00.370 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:00.372110 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-20T15:13:00.812 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:00.813350 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-20T15:13:00.812 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:00.813382 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-20T15:13:00.812 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:00.813391 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-20T15:13:01.269 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:01.270691 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-20T15:13:01.302 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:01.303820 I | op-mgr: start running mgr 2022-03-20T15:13:01.302 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:01.303854 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-20T15:13:01.564 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd stat -f json 2022-03-20T15:13:01.813 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:01.815462 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-20T15:13:01.825 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:01.826897 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-20T15:13:01.828 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:01.830696 I | op-mgr: setting services to point to mgr "a" 2022-03-20T15:13:02.082 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:13:02.106 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":3,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-20T15:13:02.107 INFO:tasks.rook: have 0/12 OSDs 2022-03-20T15:13:02.383 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:02.385299 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-20T15:13:02.384 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:02.385396 I | op-mgr: successful modules: balancer 2022-03-20T15:13:02.397 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:02.399338 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-20T15:13:02.397 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:02.399375 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-20T15:13:02.450 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:02.452397 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-20T15:13:02.549 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:02.550801 I | op-osd: creating 1 new PVCs for device set "smithi006-lv-1" 2022-03-20T15:13:02.753 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:02.755075 I | op-osd: successfully provisioned PVC "smithi006-lv-1-data-0cmfzx" 2022-03-20T15:13:02.754 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:02.755112 I | op-osd: creating 1 new PVCs for device set "smithi006-lv-2" 2022-03-20T15:13:02.952 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:02.953669 I | op-osd: successfully provisioned PVC "smithi006-lv-2-data-0hwbjq" 2022-03-20T15:13:02.952 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:02.953705 I | op-osd: creating 1 new PVCs for device set "smithi006-lv-3" 2022-03-20T15:13:03.151 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:03.153258 I | op-osd: successfully provisioned PVC "smithi006-lv-3-data-07dd5g" 2022-03-20T15:13:03.152 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:03.153298 I | op-osd: creating 1 new PVCs for device set "smithi006-lv-4" 2022-03-20T15:13:03.350 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:03.351663 I | op-osd: successfully provisioned PVC "smithi006-lv-4-data-0qbdz5" 2022-03-20T15:13:03.350 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:03.351700 I | op-osd: creating 1 new PVCs for device set "smithi045-lv-1" 2022-03-20T15:13:03.551 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:03.552513 I | op-osd: successfully provisioned PVC "smithi045-lv-1-data-0rdlmk" 2022-03-20T15:13:03.551 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:03.552547 I | op-osd: creating 1 new PVCs for device set "smithi045-lv-2" 2022-03-20T15:13:03.645 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:03.646662 I | op-mgr: successful modules: dashboard 2022-03-20T15:13:03.649 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:03.651329 I | op-mgr: successful modules: prometheus 2022-03-20T15:13:03.651 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:03.651991 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-20T15:13:03.750 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:03.752230 I | op-osd: successfully provisioned PVC "smithi045-lv-2-data-0lwhr2" 2022-03-20T15:13:03.751 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:03.752263 I | op-osd: creating 1 new PVCs for device set "smithi045-lv-3" 2022-03-20T15:13:03.952 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:03.953506 I | op-osd: successfully provisioned PVC "smithi045-lv-3-data-0jqzrc" 2022-03-20T15:13:03.952 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:03.953545 I | op-osd: creating 1 new PVCs for device set "smithi045-lv-4" 2022-03-20T15:13:04.152 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:04.153424 I | op-osd: successfully provisioned PVC "smithi045-lv-4-data-026444" 2022-03-20T15:13:04.152 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:04.153458 I | op-osd: creating 1 new PVCs for device set "smithi123-lv-1" 2022-03-20T15:13:04.352 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:04.353353 I | op-osd: successfully provisioned PVC "smithi123-lv-1-data-0vz4f8" 2022-03-20T15:13:04.352 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:04.353394 I | op-osd: creating 1 new PVCs for device set "smithi123-lv-2" 2022-03-20T15:13:04.552 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:04.553250 I | op-osd: successfully provisioned PVC "smithi123-lv-2-data-0l58zl" 2022-03-20T15:13:04.552 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:04.553286 I | op-osd: creating 1 new PVCs for device set "smithi123-lv-3" 2022-03-20T15:13:04.751 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:04.752239 I | op-osd: successfully provisioned PVC "smithi123-lv-3-data-0crj7j" 2022-03-20T15:13:04.751 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:04.752274 I | op-osd: creating 1 new PVCs for device set "smithi123-lv-4" 2022-03-20T15:13:04.951 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:04.952778 I | op-osd: successfully provisioned PVC "smithi123-lv-4-data-0vfnsc" 2022-03-20T15:13:04.961 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:04.963042 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-1-data-0cmfzx" 2022-03-20T15:13:05.239 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:05.241026 I | op-mgr: successful modules: orchestrator modules 2022-03-20T15:13:05.383 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:05.384375 I | op-osd: started OSD provisioning job for PVC "smithi006-lv-1-data-0cmfzx" 2022-03-20T15:13:05.383 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:05.384410 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-2-data-0hwbjq" 2022-03-20T15:13:05.823 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:05.824700 I | op-osd: started OSD provisioning job for PVC "smithi006-lv-2-data-0hwbjq" 2022-03-20T15:13:05.823 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:05.824734 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-3-data-07dd5g" 2022-03-20T15:13:06.232 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:06.233450 I | op-osd: started OSD provisioning job for PVC "smithi006-lv-3-data-07dd5g" 2022-03-20T15:13:06.232 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:06.233484 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-4-data-0qbdz5" 2022-03-20T15:13:06.625 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:06.627217 I | op-osd: started OSD provisioning job for PVC "smithi006-lv-4-data-0qbdz5" 2022-03-20T15:13:06.626 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:06.627254 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-1-data-0rdlmk" 2022-03-20T15:13:07.610 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:07.608694 I | op-osd: started OSD provisioning job for PVC "smithi045-lv-1-data-0rdlmk" 2022-03-20T15:13:07.610 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:07.608731 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-2-data-0lwhr2" 2022-03-20T15:13:08.422 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:08.423299 I | op-osd: started OSD provisioning job for PVC "smithi045-lv-2-data-0lwhr2" 2022-03-20T15:13:08.422 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:08.423336 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-3-data-0jqzrc" 2022-03-20T15:13:09.572 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:09.573802 I | op-osd: started OSD provisioning job for PVC "smithi045-lv-3-data-0jqzrc" 2022-03-20T15:13:09.573 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:09.573857 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-4-data-026444" 2022-03-20T15:13:10.482 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:10.483342 I | op-osd: started OSD provisioning job for PVC "smithi045-lv-4-data-026444" 2022-03-20T15:13:10.483 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:10.483426 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-1-data-0vz4f8" 2022-03-20T15:13:11.622 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:11.623613 I | op-osd: started OSD provisioning job for PVC "smithi123-lv-1-data-0vz4f8" 2022-03-20T15:13:11.622 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:11.623647 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-2-data-0l58zl" 2022-03-20T15:13:11.844 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:11.845663 I | op-osd: started OSD provisioning job for PVC "smithi123-lv-2-data-0l58zl" 2022-03-20T15:13:11.845 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:11.845699 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-3-data-0crj7j" 2022-03-20T15:13:12.108 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd stat -f json 2022-03-20T15:13:12.198 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:12.198896 I | op-osd: started OSD provisioning job for PVC "smithi123-lv-3-data-0crj7j" 2022-03-20T15:13:12.198 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:12.198931 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-4-data-0vfnsc" 2022-03-20T15:13:12.700 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:12.699801 I | op-osd: started OSD provisioning job for PVC "smithi123-lv-4-data-0vfnsc" 2022-03-20T15:13:12.700 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:12.699841 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-20T15:13:12.700 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:12.699850 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-20T15:13:12.821 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:12.817225 I | op-osd: OSD orchestration status for PVC smithi006-lv-1-data-0cmfzx is "starting" 2022-03-20T15:13:12.821 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:12.817256 I | op-osd: OSD orchestration status for PVC smithi006-lv-2-data-0hwbjq is "starting" 2022-03-20T15:13:12.822 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:12.817272 I | op-osd: OSD orchestration status for PVC smithi006-lv-3-data-07dd5g is "starting" 2022-03-20T15:13:12.822 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:12.817281 I | op-osd: OSD orchestration status for PVC smithi006-lv-4-data-0qbdz5 is "starting" 2022-03-20T15:13:12.822 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:12.817294 I | op-osd: OSD orchestration status for PVC smithi045-lv-1-data-0rdlmk is "starting" 2022-03-20T15:13:12.822 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:12.817305 I | op-osd: OSD orchestration status for PVC smithi045-lv-2-data-0lwhr2 is "starting" 2022-03-20T15:13:12.823 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:12.817319 I | op-osd: OSD orchestration status for PVC smithi045-lv-3-data-0jqzrc is "starting" 2022-03-20T15:13:12.823 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:12.817332 I | op-osd: OSD orchestration status for PVC smithi045-lv-4-data-026444 is "starting" 2022-03-20T15:13:12.823 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:12.817340 I | op-osd: OSD orchestration status for PVC smithi123-lv-1-data-0vz4f8 is "starting" 2022-03-20T15:13:12.823 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:12.817353 I | op-osd: OSD orchestration status for PVC smithi123-lv-2-data-0l58zl is "starting" 2022-03-20T15:13:12.824 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:12.817364 I | op-osd: OSD orchestration status for PVC smithi123-lv-3-data-0crj7j is "starting" 2022-03-20T15:13:12.824 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:12.817376 I | op-osd: OSD orchestration status for PVC smithi123-lv-4-data-0vfnsc is "starting" 2022-03-20T15:13:13.769 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:13:13.791 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":3,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-20T15:13:13.791 INFO:tasks.rook: have 0/12 OSDs 2022-03-20T15:13:23.793 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd stat -f json 2022-03-20T15:13:24.471 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:24.472229 I | op-osd: OSD orchestration status for node smithi045-lv-1-data-0rdlmk is "orchestrating" 2022-03-20T15:13:25.266 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:13:25.290 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":3,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-20T15:13:25.291 INFO:tasks.rook: have 0/12 OSDs 2022-03-20T15:13:25.337 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:25.338377 I | op-osd: OSD orchestration status for PVC smithi045-lv-1-data-0rdlmk is "orchestrating" 2022-03-20T15:13:25.871 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:25.872331 I | op-osd: OSD orchestration status for node smithi006-lv-3-data-07dd5g is "orchestrating" 2022-03-20T15:13:27.005 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:27.006056 I | op-osd: OSD orchestration status for PVC smithi006-lv-3-data-07dd5g is "orchestrating" 2022-03-20T15:13:27.297 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:27.297844 I | op-osd: OSD orchestration status for node smithi123-lv-1-data-0vz4f8 is "orchestrating" 2022-03-20T15:13:27.978 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:27.978911 I | op-osd: OSD orchestration status for node smithi045-lv-2-data-0lwhr2 is "orchestrating" 2022-03-20T15:13:28.294 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:28.294964 I | op-osd: OSD orchestration status for PVC smithi123-lv-1-data-0vz4f8 is "orchestrating" 2022-03-20T15:13:29.069 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:29.069994 I | op-osd: OSD orchestration status for PVC smithi045-lv-2-data-0lwhr2 is "orchestrating" 2022-03-20T15:13:29.167 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:29.168114 I | op-osd: OSD orchestration status for node smithi006-lv-4-data-0qbdz5 is "orchestrating" 2022-03-20T15:13:30.184 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:30.184597 I | op-osd: OSD orchestration status for PVC smithi006-lv-4-data-0qbdz5 is "orchestrating" 2022-03-20T15:13:30.344 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:30.345283 I | op-osd: OSD orchestration status for node smithi006-lv-1-data-0cmfzx is "orchestrating" 2022-03-20T15:13:30.585 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:30.559649 I | op-osd: OSD orchestration status for node smithi045-lv-4-data-026444 is "orchestrating" 2022-03-20T15:13:30.585 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:30.559685 I | op-osd: OSD orchestration status for node smithi045-lv-3-data-0jqzrc is "orchestrating" 2022-03-20T15:13:30.670 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:30.671321 I | op-osd: OSD orchestration status for node smithi006-lv-2-data-0hwbjq is "orchestrating" 2022-03-20T15:13:30.813 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:30.813755 I | op-osd: OSD orchestration status for PVC smithi006-lv-1-data-0cmfzx is "orchestrating" 2022-03-20T15:13:31.024 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:31.025045 I | op-osd: OSD orchestration status for PVC smithi045-lv-4-data-026444 is "orchestrating" 2022-03-20T15:13:31.063 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:31.063499 I | op-osd: OSD orchestration status for PVC smithi045-lv-3-data-0jqzrc is "orchestrating" 2022-03-20T15:13:31.085 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:31.085956 I | op-osd: OSD orchestration status for PVC smithi006-lv-2-data-0hwbjq is "orchestrating" 2022-03-20T15:13:32.220 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:32.221194 I | op-osd: OSD orchestration status for node smithi123-lv-4-data-0vfnsc is "orchestrating" 2022-03-20T15:13:32.272 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:32.272735 I | op-osd: OSD orchestration status for node smithi123-lv-3-data-0crj7j is "orchestrating" 2022-03-20T15:13:32.298 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:32.298556 I | op-osd: OSD orchestration status for node smithi123-lv-2-data-0l58zl is "orchestrating" 2022-03-20T15:13:32.487 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:32.479053 I | op-osd: OSD orchestration status for PVC smithi045-lv-3-data-0jqzrc is "failed" 2022-03-20T15:13:32.487 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:32.479093 E | op-osd: failed to provision OSD(s) on PVC smithi045-lv-3-data-0jqzrc. &{OSDs:[] Status:failed PvcBackedOSD:true Message:failed to configure devices: failed to initialize devices on PVC: failed to run ceph-volume. Running command: /usr/bin/ceph-authtool --gen-print-key 2022-03-20T15:13:32.488 INFO:tasks.rook.operator.smithi006.stdout:Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 89c05dd8-66c5-4120-bcd9-36717b1f4933 2022-03-20T15:13:32.488 INFO:tasks.rook.operator.smithi006.stdout: stderr: unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-20T15:13:32.488 INFO:tasks.rook.operator.smithi006.stdout: stderr: 2022-03-20T15:13:32.431+0000 7fd4db112700 -1 failed for service _ceph-mon._tcp 2022-03-20T15:13:32.488 INFO:tasks.rook.operator.smithi006.stdout: stderr: 2022-03-20T15:13:32.431+0000 7fd4db112700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-20T15:13:32.489 INFO:tasks.rook.operator.smithi006.stdout: stderr: [errno 2] RADOS object not found (error connecting to the cluster) 2022-03-20T15:13:32.489 INFO:tasks.rook.operator.smithi006.stdout:Traceback (most recent call last): 2022-03-20T15:13:32.489 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/sbin/ceph-volume", line 11, in 2022-03-20T15:13:32.489 INFO:tasks.rook.operator.smithi006.stdout: load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2022-03-20T15:13:32.489 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2022-03-20T15:13:32.490 INFO:tasks.rook.operator.smithi006.stdout: self.main(self.argv) 2022-03-20T15:13:32.490 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2022-03-20T15:13:32.490 INFO:tasks.rook.operator.smithi006.stdout: return f(*a, **kw) 2022-03-20T15:13:32.490 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2022-03-20T15:13:32.491 INFO:tasks.rook.operator.smithi006.stdout: terminal.dispatch(self.mapper, subcommand_args) 2022-03-20T15:13:32.491 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-20T15:13:32.491 INFO:tasks.rook.operator.smithi006.stdout: instance.main() 2022-03-20T15:13:32.491 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main 2022-03-20T15:13:32.491 INFO:tasks.rook.operator.smithi006.stdout: terminal.dispatch(self.mapper, self.argv) 2022-03-20T15:13:32.492 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-20T15:13:32.492 INFO:tasks.rook.operator.smithi006.stdout: instance.main() 2022-03-20T15:13:32.492 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main 2022-03-20T15:13:32.492 INFO:tasks.rook.operator.smithi006.stdout: self.safe_prepare(self.args) 2022-03-20T15:13:32.493 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-03-20T15:13:32.493 INFO:tasks.rook.operator.smithi006.stdout: self.prepare() 2022-03-20T15:13:32.493 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-03-20T15:13:32.493 INFO:tasks.rook.operator.smithi006.stdout: return func(*a, **kw) 2022-03-20T15:13:32.493 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-03-20T15:13:32.494 INFO:tasks.rook.operator.smithi006.stdout: osd_fsid, json.dumps(secrets)) 2022-03-20T15:13:32.494 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-03-20T15:13:32.494 INFO:tasks.rook.operator.smithi006.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-03-20T15:13:32.494 INFO:tasks.rook.operator.smithi006.stdout:RuntimeError: Unable to create a new OSD id. debug logs below: 2022-03-20T15:13:32.494 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,926][ceph_volume.main][INFO ] Running command: ceph-volume --log-path /var/log/ceph/smithi045-lv-3-data-0jqzrc raw prepare --bluestore --data /dev/mapper/vg_nvme-lv_3 2022-03-20T15:13:32.495 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,927][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk -plno KNAME,NAME,TYPE 2022-03-20T15:13:32.495 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,937][ceph_volume.process][INFO ] stdout /dev/loop0 /dev/loop0 loop 2022-03-20T15:13:32.495 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,937][ceph_volume.process][INFO ] stdout /dev/loop1 /dev/loop1 loop 2022-03-20T15:13:32.495 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,937][ceph_volume.process][INFO ] stdout /dev/loop2 /dev/loop2 loop 2022-03-20T15:13:32.496 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,937][ceph_volume.process][INFO ] stdout /dev/loop3 /dev/loop3 loop 2022-03-20T15:13:32.496 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,937][ceph_volume.process][INFO ] stdout /dev/sda /dev/sda disk 2022-03-20T15:13:32.496 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,937][ceph_volume.process][INFO ] stdout /dev/sda1 /dev/sda1 part 2022-03-20T15:13:32.496 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,937][ceph_volume.process][INFO ] stdout /dev/dm-0 /dev/mapper/vg_nvme-lv_1 lvm 2022-03-20T15:13:32.496 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,937][ceph_volume.process][INFO ] stdout /dev/dm-1 /dev/mapper/vg_nvme-lv_2 lvm 2022-03-20T15:13:32.497 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,937][ceph_volume.process][INFO ] stdout /dev/dm-2 /dev/mapper/vg_nvme-lv_3 lvm 2022-03-20T15:13:32.497 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,937][ceph_volume.process][INFO ] stdout /dev/dm-3 /dev/mapper/vg_nvme-lv_4 lvm 2022-03-20T15:13:32.497 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,937][ceph_volume.process][INFO ] stdout /dev/dm-4 /dev/mapper/vg_nvme-lv_5 lvm 2022-03-20T15:13:32.497 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,937][ceph_volume.process][INFO ] stdout /dev/nvme0n1 /dev/nvme0n1 disk 2022-03-20T15:13:32.498 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,937][ceph_volume.process][INFO ] stdout /dev/nvme1n1 /dev/nvme1n1 disk 2022-03-20T15:13:32.498 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,937][ceph_volume.process][INFO ] stdout /dev/nvme2n1 /dev/nvme2n1 disk 2022-03-20T15:13:32.498 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,938][ceph_volume.process][INFO ] stdout /dev/nvme3n1 /dev/nvme3n1 disk 2022-03-20T15:13:32.498 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,938][ceph_volume.process][INFO ] stdout /dev/nvme4n1 /dev/nvme4n1 disk 2022-03-20T15:13:32.499 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,949][ceph_volume.util.system][WARNING] Executable lvs not found on the host, will return lvs as-is 2022-03-20T15:13:32.499 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,950][ceph_volume.process][INFO ] Running command: lvs --noheadings --readonly --separator=";" -a --units=b --nosuffix -S lv_path=/dev/mapper/vg_nvme-lv_3 -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size 2022-03-20T15:13:32.499 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,251][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk --nodeps -P -o NAME,KNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL /dev/mapper/vg_nvme-lv_3 2022-03-20T15:13:32.499 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,256][ceph_volume.process][INFO ] stdout NAME="vg_nvme-lv_3" KNAME="dm-2" MAJ:MIN="253:2" FSTYPE="" MOUNTPOINT="" LABEL="" UUID="" RO="0" RM="0" MODEL="" SIZE="89.4G" STATE="running" OWNER="root" GROUP="disk" MODE="brw-rw----" ALIGNMENT="0" PHY-SEC="512" LOG-SEC="512" ROTA="0" SCHED="" TYPE="lvm" DISC-ALN="0" DISC-GRAN="512B" DISC-MAX="2T" DISC-ZERO="0" PKNAME="" PARTLABEL="" 2022-03-20T15:13:32.499 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,257][ceph_volume.process][INFO ] Running command: /usr/sbin/blkid -c /dev/null -p /dev/mapper/vg_nvme-lv_3 2022-03-20T15:13:32.500 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,261][ceph_volume.util.disk][INFO ] opening device /dev/mapper/vg_nvme-lv_3 to check for BlueStore label 2022-03-20T15:13:32.500 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,261][ceph_volume.util.disk][INFO ] opening device /dev/mapper/vg_nvme-lv_3 to check for BlueStore label 2022-03-20T15:13:32.500 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,262][ceph_volume.process][INFO ] Running command: /usr/sbin/udevadm info --query=property /dev/mapper/vg_nvme-lv_3 2022-03-20T15:13:32.500 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,270][ceph_volume.process][INFO ] stdout DEVLINKS=/dev/disk/by-id/dm-uuid-LVM-ce31WZSvS0kvk6a73YxzJD4NdAMzmzV71SYBD8Ty1GaH36y2Kv1lDEBNUTJpC8Kw /dev/vg_nvme/lv_3 /dev/disk/by-id/dm-name-vg_nvme-lv_3 /dev/mapper/vg_nvme-lv_3 2022-03-20T15:13:32.501 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,270][ceph_volume.process][INFO ] stdout DEVNAME=/dev/dm-2 2022-03-20T15:13:32.501 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,270][ceph_volume.process][INFO ] stdout DEVPATH=/devices/virtual/block/dm-2 2022-03-20T15:13:32.501 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,270][ceph_volume.process][INFO ] stdout DEVTYPE=disk 2022-03-20T15:13:32.501 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,270][ceph_volume.process][INFO ] stdout DM_LV_NAME=lv_3 2022-03-20T15:13:32.501 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,270][ceph_volume.process][INFO ] stdout DM_NAME=vg_nvme-lv_3 2022-03-20T15:13:32.502 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,270][ceph_volume.process][INFO ] stdout DM_SUSPENDED=0 2022-03-20T15:13:32.502 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,270][ceph_volume.process][INFO ] stdout DM_UDEV_DISABLE_LIBRARY_FALLBACK_FLAG=1 2022-03-20T15:13:32.502 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,270][ceph_volume.process][INFO ] stdout DM_UDEV_PRIMARY_SOURCE_FLAG=1 2022-03-20T15:13:32.502 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,270][ceph_volume.process][INFO ] stdout DM_UDEV_RULES=1 2022-03-20T15:13:32.503 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,270][ceph_volume.process][INFO ] stdout DM_UDEV_RULES_VSN=2 2022-03-20T15:13:32.503 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,270][ceph_volume.process][INFO ] stdout DM_UUID=LVM-ce31WZSvS0kvk6a73YxzJD4NdAMzmzV71SYBD8Ty1GaH36y2Kv1lDEBNUTJpC8Kw 2022-03-20T15:13:32.503 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,270][ceph_volume.process][INFO ] stdout DM_VG_NAME=vg_nvme 2022-03-20T15:13:32.503 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,270][ceph_volume.process][INFO ] stdout MAJOR=253 2022-03-20T15:13:32.503 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,270][ceph_volume.process][INFO ] stdout MINOR=2 2022-03-20T15:13:32.504 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,270][ceph_volume.process][INFO ] stdout SUBSYSTEM=block 2022-03-20T15:13:32.504 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,271][ceph_volume.process][INFO ] stdout TAGS=:systemd: 2022-03-20T15:13:32.504 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,271][ceph_volume.process][INFO ] stdout USEC_INITIALIZED=225190740 2022-03-20T15:13:32.504 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,271][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-authtool --gen-print-key 2022-03-20T15:13:32.504 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,287][ceph_volume.process][INFO ] stdout AQCcRDdi0IcNERAATrotPEVMD5pnNBpMm46YCw== 2022-03-20T15:13:32.505 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,288][ceph_volume.process][INFO ] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 89c05dd8-66c5-4120-bcd9-36717b1f4933 2022-03-20T15:13:32.505 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,435][ceph_volume.process][INFO ] stderr unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-20T15:13:32.505 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,435][ceph_volume.process][INFO ] stderr 2022-03-20T15:13:32.431+0000 7fd4db112700 -1 failed for service _ceph-mon._tcp 2022-03-20T15:13:32.505 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,435][ceph_volume.process][INFO ] stderr 2022-03-20T15:13:32.431+0000 7fd4db112700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-20T15:13:32.506 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,436][ceph_volume.process][INFO ] stderr [errno 2] RADOS object not found (error connecting to the cluster) 2022-03-20T15:13:32.506 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,436][ceph_volume.devices.raw.prepare][ERROR ] raw prepare was unable to complete 2022-03-20T15:13:32.506 INFO:tasks.rook.operator.smithi006.stdout:Traceback (most recent call last): 2022-03-20T15:13:32.506 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-03-20T15:13:32.507 INFO:tasks.rook.operator.smithi006.stdout: self.prepare() 2022-03-20T15:13:32.507 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-03-20T15:13:32.507 INFO:tasks.rook.operator.smithi006.stdout: return func(*a, **kw) 2022-03-20T15:13:32.507 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-03-20T15:13:32.507 INFO:tasks.rook.operator.smithi006.stdout: osd_fsid, json.dumps(secrets)) 2022-03-20T15:13:32.508 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-03-20T15:13:32.508 INFO:tasks.rook.operator.smithi006.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-03-20T15:13:32.508 INFO:tasks.rook.operator.smithi006.stdout:RuntimeError: Unable to create a new OSD id 2022-03-20T15:13:32.508 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,437][ceph_volume.devices.raw.prepare][INFO ] will rollback OSD ID creation 2022-03-20T15:13:32.509 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,437][ceph_volume][ERROR ] exception caught by decorator 2022-03-20T15:13:32.509 INFO:tasks.rook.operator.smithi006.stdout:Traceback (most recent call last): 2022-03-20T15:13:32.509 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2022-03-20T15:13:32.509 INFO:tasks.rook.operator.smithi006.stdout: return f(*a, **kw) 2022-03-20T15:13:32.509 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2022-03-20T15:13:32.510 INFO:tasks.rook.operator.smithi006.stdout: terminal.dispatch(self.mapper, subcommand_args) 2022-03-20T15:13:32.510 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-20T15:13:32.510 INFO:tasks.rook.operator.smithi006.stdout: instance.main() 2022-03-20T15:13:32.510 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main 2022-03-20T15:13:32.510 INFO:tasks.rook.operator.smithi006.stdout: terminal.dispatch(self.mapper, self.argv) 2022-03-20T15:13:32.511 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-20T15:13:32.511 INFO:tasks.rook.operator.smithi006.stdout: instance.main() 2022-03-20T15:13:32.511 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main 2022-03-20T15:13:32.511 INFO:tasks.rook.operator.smithi006.stdout: self.safe_prepare(self.args) 2022-03-20T15:13:32.512 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-03-20T15:13:32.512 INFO:tasks.rook.operator.smithi006.stdout: self.prepare() 2022-03-20T15:13:32.512 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-03-20T15:13:32.512 INFO:tasks.rook.operator.smithi006.stdout: return func(*a, **kw) 2022-03-20T15:13:32.512 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-03-20T15:13:32.513 INFO:tasks.rook.operator.smithi006.stdout: osd_fsid, json.dumps(secrets)) 2022-03-20T15:13:32.513 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-03-20T15:13:32.513 INFO:tasks.rook.operator.smithi006.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-03-20T15:13:32.513 INFO:tasks.rook.operator.smithi006.stdout:RuntimeError: Unable to create a new OSD id 2022-03-20T15:13:32.513 INFO:tasks.rook.operator.smithi006.stdout:: exit status 1} 2022-03-20T15:13:32.652 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:32.652982 I | op-osd: OSD orchestration status for PVC smithi123-lv-3-data-0crj7j is "orchestrating" 2022-03-20T15:13:32.716 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:32.716640 I | op-osd: OSD orchestration status for PVC smithi123-lv-2-data-0l58zl is "orchestrating" 2022-03-20T15:13:32.721 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:32.722241 I | op-osd: OSD orchestration status for PVC smithi123-lv-4-data-0vfnsc is "orchestrating" 2022-03-20T15:13:32.870 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:32.870388 I | op-osd: OSD orchestration status for PVC smithi045-lv-1-data-0rdlmk is "completed" 2022-03-20T15:13:32.870 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:32.870418 I | op-osd: creating OSD 0 on PVC "smithi045-lv-1-data-0rdlmk" 2022-03-20T15:13:32.870 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:32.870431 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-1-data-0rdlmk" 2022-03-20T15:13:34.453 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:34.450306 I | op-osd: OSD orchestration status for PVC smithi123-lv-3-data-0crj7j is "failed" 2022-03-20T15:13:34.454 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:34.450354 E | op-osd: failed to provision OSD(s) on PVC smithi123-lv-3-data-0crj7j. &{OSDs:[] Status:failed PvcBackedOSD:true Message:failed to configure devices: failed to initialize devices on PVC: failed to run ceph-volume. Running command: /usr/bin/ceph-authtool --gen-print-key 2022-03-20T15:13:34.454 INFO:tasks.rook.operator.smithi006.stdout:Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4c3a0862-b08a-4b00-b105-3c1e58b3fe99 2022-03-20T15:13:34.454 INFO:tasks.rook.operator.smithi006.stdout: stderr: unable to get monitor info from DNS SRV with service name: 2022-03-20T15:13:34.391+0000 7ffb6611b700 -1 failed for service _ceph-mon._tcp 2022-03-20T15:13:34.454 INFO:tasks.rook.operator.smithi006.stdout: stderr: ceph-mon 2022-03-20T15:13:34.455 INFO:tasks.rook.operator.smithi006.stdout: stderr: 2022-03-20T15:13:34.391+0000 7ffb6611b700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-20T15:13:34.455 INFO:tasks.rook.operator.smithi006.stdout: stderr: [errno 2] RADOS object not found (error connecting to the cluster) 2022-03-20T15:13:34.455 INFO:tasks.rook.operator.smithi006.stdout:Traceback (most recent call last): 2022-03-20T15:13:34.455 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/sbin/ceph-volume", line 11, in 2022-03-20T15:13:34.456 INFO:tasks.rook.operator.smithi006.stdout: load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2022-03-20T15:13:34.456 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2022-03-20T15:13:34.456 INFO:tasks.rook.operator.smithi006.stdout: self.main(self.argv) 2022-03-20T15:13:34.456 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2022-03-20T15:13:34.456 INFO:tasks.rook.operator.smithi006.stdout: return f(*a, **kw) 2022-03-20T15:13:34.457 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2022-03-20T15:13:34.457 INFO:tasks.rook.operator.smithi006.stdout: terminal.dispatch(self.mapper, subcommand_args) 2022-03-20T15:13:34.457 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-20T15:13:34.457 INFO:tasks.rook.operator.smithi006.stdout: instance.main() 2022-03-20T15:13:34.458 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main 2022-03-20T15:13:34.458 INFO:tasks.rook.operator.smithi006.stdout: terminal.dispatch(self.mapper, self.argv) 2022-03-20T15:13:34.458 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-20T15:13:34.458 INFO:tasks.rook.operator.smithi006.stdout: instance.main() 2022-03-20T15:13:34.459 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main 2022-03-20T15:13:34.459 INFO:tasks.rook.operator.smithi006.stdout: self.safe_prepare(self.args) 2022-03-20T15:13:34.459 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-03-20T15:13:34.459 INFO:tasks.rook.operator.smithi006.stdout: self.prepare() 2022-03-20T15:13:34.460 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-03-20T15:13:34.460 INFO:tasks.rook.operator.smithi006.stdout: return func(*a, **kw) 2022-03-20T15:13:34.460 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-03-20T15:13:34.460 INFO:tasks.rook.operator.smithi006.stdout: osd_fsid, json.dumps(secrets)) 2022-03-20T15:13:34.461 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-03-20T15:13:34.468 INFO:tasks.rook.operator.smithi006.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-03-20T15:13:34.469 INFO:tasks.rook.operator.smithi006.stdout:RuntimeError: Unable to create a new OSD id. debug logs below: 2022-03-20T15:13:34.469 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,014][ceph_volume.main][INFO ] Running command: ceph-volume --log-path /var/log/ceph/smithi123-lv-3-data-0crj7j raw prepare --bluestore --data /dev/mapper/vg_nvme-lv_3 2022-03-20T15:13:34.469 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,014][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk -plno KNAME,NAME,TYPE 2022-03-20T15:13:34.469 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,024][ceph_volume.process][INFO ] stdout /dev/loop0 /dev/loop0 loop 2022-03-20T15:13:34.470 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,024][ceph_volume.process][INFO ] stdout /dev/loop1 /dev/loop1 loop 2022-03-20T15:13:34.470 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,024][ceph_volume.process][INFO ] stdout /dev/loop2 /dev/loop2 loop 2022-03-20T15:13:34.470 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,024][ceph_volume.process][INFO ] stdout /dev/loop3 /dev/loop3 loop 2022-03-20T15:13:34.470 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,024][ceph_volume.process][INFO ] stdout /dev/sda /dev/sda disk 2022-03-20T15:13:34.471 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,024][ceph_volume.process][INFO ] stdout /dev/sda1 /dev/sda1 part 2022-03-20T15:13:34.471 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,024][ceph_volume.process][INFO ] stdout /dev/dm-0 /dev/mapper/vg_nvme-lv_1 lvm 2022-03-20T15:13:34.471 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,024][ceph_volume.process][INFO ] stdout /dev/dm-1 /dev/mapper/vg_nvme-lv_2 lvm 2022-03-20T15:13:34.471 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,024][ceph_volume.process][INFO ] stdout /dev/dm-2 /dev/mapper/vg_nvme-lv_3 lvm 2022-03-20T15:13:34.472 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,025][ceph_volume.process][INFO ] stdout /dev/dm-3 /dev/mapper/vg_nvme-lv_4 lvm 2022-03-20T15:13:34.472 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,025][ceph_volume.process][INFO ] stdout /dev/dm-4 /dev/mapper/vg_nvme-lv_5 lvm 2022-03-20T15:13:34.472 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,025][ceph_volume.process][INFO ] stdout /dev/nvme0n1 /dev/nvme0n1 disk 2022-03-20T15:13:34.472 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,025][ceph_volume.process][INFO ] stdout /dev/nvme1n1 /dev/nvme1n1 disk 2022-03-20T15:13:34.472 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,025][ceph_volume.process][INFO ] stdout /dev/nvme2n1 /dev/nvme2n1 disk 2022-03-20T15:13:34.473 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,025][ceph_volume.process][INFO ] stdout /dev/nvme3n1 /dev/nvme3n1 disk 2022-03-20T15:13:34.473 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,025][ceph_volume.process][INFO ] stdout /dev/nvme4n1 /dev/nvme4n1 disk 2022-03-20T15:13:34.473 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,035][ceph_volume.util.system][WARNING] Executable lvs not found on the host, will return lvs as-is 2022-03-20T15:13:34.473 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,035][ceph_volume.process][INFO ] Running command: lvs --noheadings --readonly --separator=";" -a --units=b --nosuffix -S lv_path=/dev/mapper/vg_nvme-lv_3 -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size 2022-03-20T15:13:34.474 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,243][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk --nodeps -P -o NAME,KNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL /dev/mapper/vg_nvme-lv_3 2022-03-20T15:13:34.474 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,248][ceph_volume.process][INFO ] stdout NAME="vg_nvme-lv_3" KNAME="dm-2" MAJ:MIN="253:2" FSTYPE="" MOUNTPOINT="" LABEL="" UUID="" RO="0" RM="0" MODEL="" SIZE="89.4G" STATE="running" OWNER="root" GROUP="disk" MODE="brw-rw----" ALIGNMENT="0" PHY-SEC="512" LOG-SEC="512" ROTA="0" SCHED="" TYPE="lvm" DISC-ALN="0" DISC-GRAN="512B" DISC-MAX="2T" DISC-ZERO="0" PKNAME="" PARTLABEL="" 2022-03-20T15:13:34.474 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,248][ceph_volume.process][INFO ] Running command: /usr/sbin/blkid -c /dev/null -p /dev/mapper/vg_nvme-lv_3 2022-03-20T15:13:34.474 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,251][ceph_volume.util.disk][INFO ] opening device /dev/mapper/vg_nvme-lv_3 to check for BlueStore label 2022-03-20T15:13:34.475 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,252][ceph_volume.util.disk][INFO ] opening device /dev/mapper/vg_nvme-lv_3 to check for BlueStore label 2022-03-20T15:13:34.475 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,252][ceph_volume.process][INFO ] Running command: /usr/sbin/udevadm info --query=property /dev/mapper/vg_nvme-lv_3 2022-03-20T15:13:34.475 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,259][ceph_volume.process][INFO ] stdout DEVLINKS=/dev/mapper/vg_nvme-lv_3 /dev/disk/by-id/dm-name-vg_nvme-lv_3 /dev/disk/by-id/dm-uuid-LVM-PZaq4q5Lcwhjnsid2JGepycLeSgZYtJ62wOvEg7V5hlog53S7OZJodacRc6Mxd1T /dev/vg_nvme/lv_3 2022-03-20T15:13:34.475 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,259][ceph_volume.process][INFO ] stdout DEVNAME=/dev/dm-2 2022-03-20T15:13:34.476 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,259][ceph_volume.process][INFO ] stdout DEVPATH=/devices/virtual/block/dm-2 2022-03-20T15:13:34.476 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,260][ceph_volume.process][INFO ] stdout DEVTYPE=disk 2022-03-20T15:13:34.476 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,260][ceph_volume.process][INFO ] stdout DM_LV_NAME=lv_3 2022-03-20T15:13:34.476 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,260][ceph_volume.process][INFO ] stdout DM_NAME=vg_nvme-lv_3 2022-03-20T15:13:34.477 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,260][ceph_volume.process][INFO ] stdout DM_SUSPENDED=0 2022-03-20T15:13:34.477 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,260][ceph_volume.process][INFO ] stdout DM_UDEV_DISABLE_LIBRARY_FALLBACK_FLAG=1 2022-03-20T15:13:34.477 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,260][ceph_volume.process][INFO ] stdout DM_UDEV_PRIMARY_SOURCE_FLAG=1 2022-03-20T15:13:34.477 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,260][ceph_volume.process][INFO ] stdout DM_UDEV_RULES=1 2022-03-20T15:13:34.478 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,260][ceph_volume.process][INFO ] stdout DM_UDEV_RULES_VSN=2 2022-03-20T15:13:34.478 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,260][ceph_volume.process][INFO ] stdout DM_UUID=LVM-PZaq4q5Lcwhjnsid2JGepycLeSgZYtJ62wOvEg7V5hlog53S7OZJodacRc6Mxd1T 2022-03-20T15:13:34.478 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,260][ceph_volume.process][INFO ] stdout DM_VG_NAME=vg_nvme 2022-03-20T15:13:34.478 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,260][ceph_volume.process][INFO ] stdout MAJOR=253 2022-03-20T15:13:34.479 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,260][ceph_volume.process][INFO ] stdout MINOR=2 2022-03-20T15:13:34.479 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,260][ceph_volume.process][INFO ] stdout SUBSYSTEM=block 2022-03-20T15:13:34.479 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,260][ceph_volume.process][INFO ] stdout TAGS=:systemd: 2022-03-20T15:13:34.479 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,260][ceph_volume.process][INFO ] stdout USEC_INITIALIZED=222914530 2022-03-20T15:13:34.480 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,261][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-authtool --gen-print-key 2022-03-20T15:13:34.480 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,281][ceph_volume.process][INFO ] stdout AQCeRDdijHCvEBAA+O3GAhABVcCaZ8CQ2t7AEQ== 2022-03-20T15:13:34.480 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,281][ceph_volume.process][INFO ] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4c3a0862-b08a-4b00-b105-3c1e58b3fe99 2022-03-20T15:13:34.480 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,397][ceph_volume.process][INFO ] stderr unable to get monitor info from DNS SRV with service name: 2022-03-20T15:13:34.391+0000 7ffb6611b700 -1 failed for service _ceph-mon._tcp 2022-03-20T15:13:34.480 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,397][ceph_volume.process][INFO ] stderr ceph-mon 2022-03-20T15:13:34.481 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,397][ceph_volume.process][INFO ] stderr 2022-03-20T15:13:34.391+0000 7ffb6611b700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-20T15:13:34.481 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,397][ceph_volume.process][INFO ] stderr [errno 2] RADOS object not found (error connecting to the cluster) 2022-03-20T15:13:34.481 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,397][ceph_volume.devices.raw.prepare][ERROR ] raw prepare was unable to complete 2022-03-20T15:13:34.481 INFO:tasks.rook.operator.smithi006.stdout:Traceback (most recent call last): 2022-03-20T15:13:34.482 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-03-20T15:13:34.482 INFO:tasks.rook.operator.smithi006.stdout: self.prepare() 2022-03-20T15:13:34.482 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-03-20T15:13:34.482 INFO:tasks.rook.operator.smithi006.stdout: return func(*a, **kw) 2022-03-20T15:13:34.483 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-03-20T15:13:34.483 INFO:tasks.rook.operator.smithi006.stdout: osd_fsid, json.dumps(secrets)) 2022-03-20T15:13:34.483 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-03-20T15:13:34.483 INFO:tasks.rook.operator.smithi006.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-03-20T15:13:34.483 INFO:tasks.rook.operator.smithi006.stdout:RuntimeError: Unable to create a new OSD id 2022-03-20T15:13:34.484 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,398][ceph_volume.devices.raw.prepare][INFO ] will rollback OSD ID creation 2022-03-20T15:13:34.484 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,398][ceph_volume][ERROR ] exception caught by decorator 2022-03-20T15:13:34.484 INFO:tasks.rook.operator.smithi006.stdout:Traceback (most recent call last): 2022-03-20T15:13:34.484 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2022-03-20T15:13:34.485 INFO:tasks.rook.operator.smithi006.stdout: return f(*a, **kw) 2022-03-20T15:13:34.485 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2022-03-20T15:13:34.485 INFO:tasks.rook.operator.smithi006.stdout: terminal.dispatch(self.mapper, subcommand_args) 2022-03-20T15:13:34.485 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-20T15:13:34.485 INFO:tasks.rook.operator.smithi006.stdout: instance.main() 2022-03-20T15:13:34.486 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main 2022-03-20T15:13:34.486 INFO:tasks.rook.operator.smithi006.stdout: terminal.dispatch(self.mapper, self.argv) 2022-03-20T15:13:34.486 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-20T15:13:34.486 INFO:tasks.rook.operator.smithi006.stdout: instance.main() 2022-03-20T15:13:34.487 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main 2022-03-20T15:13:34.487 INFO:tasks.rook.operator.smithi006.stdout: self.safe_prepare(self.args) 2022-03-20T15:13:34.487 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-03-20T15:13:34.487 INFO:tasks.rook.operator.smithi006.stdout: self.prepare() 2022-03-20T15:13:34.488 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-03-20T15:13:34.488 INFO:tasks.rook.operator.smithi006.stdout: return func(*a, **kw) 2022-03-20T15:13:34.488 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-03-20T15:13:34.488 INFO:tasks.rook.operator.smithi006.stdout: osd_fsid, json.dumps(secrets)) 2022-03-20T15:13:34.488 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-03-20T15:13:34.489 INFO:tasks.rook.operator.smithi006.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-03-20T15:13:34.489 INFO:tasks.rook.operator.smithi006.stdout:RuntimeError: Unable to create a new OSD id 2022-03-20T15:13:34.489 INFO:tasks.rook.operator.smithi006.stdout:: exit status 1} 2022-03-20T15:13:35.292 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd stat -f json 2022-03-20T15:13:35.623 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:35.623065 I | op-osd: OSD orchestration status for PVC smithi006-lv-3-data-07dd5g is "completed" 2022-03-20T15:13:35.623 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:35.623091 I | op-osd: creating OSD 1 on PVC "smithi006-lv-3-data-07dd5g" 2022-03-20T15:13:35.623 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:35.623100 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-3-data-07dd5g" 2022-03-20T15:13:35.863 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:13:35.881 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":11,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1647789214,"num_remapped_pgs":0} 2022-03-20T15:13:35.882 INFO:tasks.rook: have 0/12 OSDs 2022-03-20T15:13:36.544 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:36.544160 I | op-osd: OSD orchestration status for PVC smithi045-lv-2-data-0lwhr2 is "completed" 2022-03-20T15:13:36.545 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:36.544201 I | op-osd: creating OSD 3 on PVC "smithi045-lv-2-data-0lwhr2" 2022-03-20T15:13:36.545 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:36.544223 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-2-data-0lwhr2" 2022-03-20T15:13:37.033 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:37.031832 I | op-osd: OSD orchestration status for PVC smithi123-lv-1-data-0vz4f8 is "completed" 2022-03-20T15:13:37.034 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:37.031852 I | op-osd: creating OSD 2 on PVC "smithi123-lv-1-data-0vz4f8" 2022-03-20T15:13:37.034 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:37.031864 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-1-data-0vz4f8" 2022-03-20T15:13:38.391 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:38.390724 I | op-osd: OSD orchestration status for node smithi123-lv-3-data-0crj7j is "orchestrating" 2022-03-20T15:13:38.399 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:38.391143 I | op-osd: OSD orchestration status for PVC smithi123-lv-3-data-0crj7j is "orchestrating" 2022-03-20T15:13:38.399 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:38.391313 I | op-osd: OSD orchestration status for node smithi045-lv-3-data-0jqzrc is "orchestrating" 2022-03-20T15:13:39.103 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:39.102843 I | op-osd: OSD orchestration status for PVC smithi006-lv-2-data-0hwbjq is "completed" 2022-03-20T15:13:39.104 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:39.102866 I | op-osd: creating OSD 4 on PVC "smithi006-lv-2-data-0hwbjq" 2022-03-20T15:13:39.104 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:39.102877 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-2-data-0hwbjq" 2022-03-20T15:13:41.433 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:41.432981 I | op-osd: OSD orchestration status for PVC smithi006-lv-1-data-0cmfzx is "completed" 2022-03-20T15:13:41.435 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:41.433013 I | op-osd: creating OSD 6 on PVC "smithi006-lv-1-data-0cmfzx" 2022-03-20T15:13:41.435 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:41.433023 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-1-data-0cmfzx" 2022-03-20T15:13:42.235 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:42.234053 I | op-osd: OSD orchestration status for PVC smithi006-lv-4-data-0qbdz5 is "completed" 2022-03-20T15:13:42.235 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:42.234089 I | op-osd: creating OSD 5 on PVC "smithi006-lv-4-data-0qbdz5" 2022-03-20T15:13:42.235 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:42.234105 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-4-data-0qbdz5" 2022-03-20T15:13:43.777 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:43.774422 I | op-osd: OSD orchestration status for PVC smithi045-lv-3-data-0jqzrc is "orchestrating" 2022-03-20T15:13:43.778 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:43.774995 I | op-osd: OSD orchestration status for PVC smithi045-lv-4-data-026444 is "completed" 2022-03-20T15:13:43.778 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:43.775031 I | op-osd: creating OSD 7 on PVC "smithi045-lv-4-data-026444" 2022-03-20T15:13:43.780 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:43.775050 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-4-data-026444" 2022-03-20T15:13:45.633 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:45.629415 I | op-osd: OSD orchestration status for PVC smithi123-lv-4-data-0vfnsc is "completed" 2022-03-20T15:13:45.633 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:45.629467 I | op-osd: creating OSD 8 on PVC "smithi123-lv-4-data-0vfnsc" 2022-03-20T15:13:45.634 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:45.629495 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-4-data-0vfnsc" 2022-03-20T15:13:45.883 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd stat -f json 2022-03-20T15:13:47.248 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:47.247407 I | op-osd: OSD orchestration status for PVC smithi123-lv-2-data-0l58zl is "completed" 2022-03-20T15:13:47.249 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:47.247450 I | op-osd: creating OSD 9 on PVC "smithi123-lv-2-data-0l58zl" 2022-03-20T15:13:47.249 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:47.247477 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-2-data-0l58zl" 2022-03-20T15:13:48.301 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:13:48.324 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":15,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1647789222,"num_remapped_pgs":0} 2022-03-20T15:13:48.324 INFO:tasks.rook: have 0/12 OSDs 2022-03-20T15:13:48.898 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:48.896549 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph osds: 2 failures encountered while running osds on nodes in namespace "rook-ceph". 2022-03-20T15:13:48.899 INFO:tasks.rook.operator.smithi006.stdout:failed to provision OSD(s) on PVC smithi045-lv-3-data-0jqzrc. &{OSDs:[] Status:failed PvcBackedOSD:true Message:failed to configure devices: failed to initialize devices on PVC: failed to run ceph-volume. Running command: /usr/bin/ceph-authtool --gen-print-key 2022-03-20T15:13:48.899 INFO:tasks.rook.operator.smithi006.stdout:Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 89c05dd8-66c5-4120-bcd9-36717b1f4933 2022-03-20T15:13:48.899 INFO:tasks.rook.operator.smithi006.stdout: stderr: unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-20T15:13:48.900 INFO:tasks.rook.operator.smithi006.stdout: stderr: 2022-03-20T15:13:32.431+0000 7fd4db112700 -1 failed for service _ceph-mon._tcp 2022-03-20T15:13:48.900 INFO:tasks.rook.operator.smithi006.stdout: stderr: 2022-03-20T15:13:32.431+0000 7fd4db112700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-20T15:13:48.900 INFO:tasks.rook.operator.smithi006.stdout: stderr: [errno 2] RADOS object not found (error connecting to the cluster) 2022-03-20T15:13:48.900 INFO:tasks.rook.operator.smithi006.stdout:Traceback (most recent call last): 2022-03-20T15:13:48.901 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/sbin/ceph-volume", line 11, in 2022-03-20T15:13:48.901 INFO:tasks.rook.operator.smithi006.stdout: load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2022-03-20T15:13:48.901 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2022-03-20T15:13:48.901 INFO:tasks.rook.operator.smithi006.stdout: self.main(self.argv) 2022-03-20T15:13:48.902 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2022-03-20T15:13:48.902 INFO:tasks.rook.operator.smithi006.stdout: return f(*a, **kw) 2022-03-20T15:13:48.902 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2022-03-20T15:13:48.902 INFO:tasks.rook.operator.smithi006.stdout: terminal.dispatch(self.mapper, subcommand_args) 2022-03-20T15:13:48.902 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-20T15:13:48.903 INFO:tasks.rook.operator.smithi006.stdout: instance.main() 2022-03-20T15:13:48.903 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main 2022-03-20T15:13:48.903 INFO:tasks.rook.operator.smithi006.stdout: terminal.dispatch(self.mapper, self.argv) 2022-03-20T15:13:48.903 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-20T15:13:48.904 INFO:tasks.rook.operator.smithi006.stdout: instance.main() 2022-03-20T15:13:48.904 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main 2022-03-20T15:13:48.904 INFO:tasks.rook.operator.smithi006.stdout: self.safe_prepare(self.args) 2022-03-20T15:13:48.904 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-03-20T15:13:48.917 INFO:tasks.rook.operator.smithi006.stdout: self.prepare() 2022-03-20T15:13:48.918 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-03-20T15:13:48.918 INFO:tasks.rook.operator.smithi006.stdout: return func(*a, **kw) 2022-03-20T15:13:48.918 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-03-20T15:13:48.918 INFO:tasks.rook.operator.smithi006.stdout: osd_fsid, json.dumps(secrets)) 2022-03-20T15:13:48.919 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-03-20T15:13:48.919 INFO:tasks.rook.operator.smithi006.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-03-20T15:13:48.919 INFO:tasks.rook.operator.smithi006.stdout:RuntimeError: Unable to create a new OSD id. debug logs below: 2022-03-20T15:13:48.920 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,926][ceph_volume.main][INFO ] Running command: ceph-volume --log-path /var/log/ceph/smithi045-lv-3-data-0jqzrc raw prepare --bluestore --data /dev/mapper/vg_nvme-lv_3 2022-03-20T15:13:48.920 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,927][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk -plno KNAME,NAME,TYPE 2022-03-20T15:13:48.920 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,937][ceph_volume.process][INFO ] stdout /dev/loop0 /dev/loop0 loop 2022-03-20T15:13:48.920 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,937][ceph_volume.process][INFO ] stdout /dev/loop1 /dev/loop1 loop 2022-03-20T15:13:48.920 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,937][ceph_volume.process][INFO ] stdout /dev/loop2 /dev/loop2 loop 2022-03-20T15:13:48.921 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,937][ceph_volume.process][INFO ] stdout /dev/loop3 /dev/loop3 loop 2022-03-20T15:13:48.921 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,937][ceph_volume.process][INFO ] stdout /dev/sda /dev/sda disk 2022-03-20T15:13:48.921 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,937][ceph_volume.process][INFO ] stdout /dev/sda1 /dev/sda1 part 2022-03-20T15:13:48.921 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,937][ceph_volume.process][INFO ] stdout /dev/dm-0 /dev/mapper/vg_nvme-lv_1 lvm 2022-03-20T15:13:48.922 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,937][ceph_volume.process][INFO ] stdout /dev/dm-1 /dev/mapper/vg_nvme-lv_2 lvm 2022-03-20T15:13:48.922 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,937][ceph_volume.process][INFO ] stdout /dev/dm-2 /dev/mapper/vg_nvme-lv_3 lvm 2022-03-20T15:13:48.922 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,937][ceph_volume.process][INFO ] stdout /dev/dm-3 /dev/mapper/vg_nvme-lv_4 lvm 2022-03-20T15:13:48.922 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,937][ceph_volume.process][INFO ] stdout /dev/dm-4 /dev/mapper/vg_nvme-lv_5 lvm 2022-03-20T15:13:48.923 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,937][ceph_volume.process][INFO ] stdout /dev/nvme0n1 /dev/nvme0n1 disk 2022-03-20T15:13:48.923 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,937][ceph_volume.process][INFO ] stdout /dev/nvme1n1 /dev/nvme1n1 disk 2022-03-20T15:13:48.923 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,937][ceph_volume.process][INFO ] stdout /dev/nvme2n1 /dev/nvme2n1 disk 2022-03-20T15:13:48.923 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,938][ceph_volume.process][INFO ] stdout /dev/nvme3n1 /dev/nvme3n1 disk 2022-03-20T15:13:48.923 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,938][ceph_volume.process][INFO ] stdout /dev/nvme4n1 /dev/nvme4n1 disk 2022-03-20T15:13:48.924 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,949][ceph_volume.util.system][WARNING] Executable lvs not found on the host, will return lvs as-is 2022-03-20T15:13:48.924 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:31,950][ceph_volume.process][INFO ] Running command: lvs --noheadings --readonly --separator=";" -a --units=b --nosuffix -S lv_path=/dev/mapper/vg_nvme-lv_3 -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size 2022-03-20T15:13:48.924 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,251][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk --nodeps -P -o NAME,KNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL /dev/mapper/vg_nvme-lv_3 2022-03-20T15:13:48.924 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,256][ceph_volume.process][INFO ] stdout NAME="vg_nvme-lv_3" KNAME="dm-2" MAJ:MIN="253:2" FSTYPE="" MOUNTPOINT="" LABEL="" UUID="" RO="0" RM="0" MODEL="" SIZE="89.4G" STATE="running" OWNER="root" GROUP="disk" MODE="brw-rw----" ALIGNMENT="0" PHY-SEC="512" LOG-SEC="512" ROTA="0" SCHED="" TYPE="lvm" DISC-ALN="0" DISC-GRAN="512B" DISC-MAX="2T" DISC-ZERO="0" PKNAME="" PARTLABEL="" 2022-03-20T15:13:48.925 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,257][ceph_volume.process][INFO ] Running command: /usr/sbin/blkid -c /dev/null -p /dev/mapper/vg_nvme-lv_3 2022-03-20T15:13:48.925 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,261][ceph_volume.util.disk][INFO ] opening device /dev/mapper/vg_nvme-lv_3 to check for BlueStore label 2022-03-20T15:13:48.925 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,261][ceph_volume.util.disk][INFO ] opening device /dev/mapper/vg_nvme-lv_3 to check for BlueStore label 2022-03-20T15:13:48.925 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,262][ceph_volume.process][INFO ] Running command: /usr/sbin/udevadm info --query=property /dev/mapper/vg_nvme-lv_3 2022-03-20T15:13:48.926 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,270][ceph_volume.process][INFO ] stdout DEVLINKS=/dev/disk/by-id/dm-uuid-LVM-ce31WZSvS0kvk6a73YxzJD4NdAMzmzV71SYBD8Ty1GaH36y2Kv1lDEBNUTJpC8Kw /dev/vg_nvme/lv_3 /dev/disk/by-id/dm-name-vg_nvme-lv_3 /dev/mapper/vg_nvme-lv_3 2022-03-20T15:13:48.926 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,270][ceph_volume.process][INFO ] stdout DEVNAME=/dev/dm-2 2022-03-20T15:13:48.926 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,270][ceph_volume.process][INFO ] stdout DEVPATH=/devices/virtual/block/dm-2 2022-03-20T15:13:48.926 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,270][ceph_volume.process][INFO ] stdout DEVTYPE=disk 2022-03-20T15:13:48.926 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,270][ceph_volume.process][INFO ] stdout DM_LV_NAME=lv_3 2022-03-20T15:13:48.927 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,270][ceph_volume.process][INFO ] stdout DM_NAME=vg_nvme-lv_3 2022-03-20T15:13:48.927 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,270][ceph_volume.process][INFO ] stdout DM_SUSPENDED=0 2022-03-20T15:13:48.927 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,270][ceph_volume.process][INFO ] stdout DM_UDEV_DISABLE_LIBRARY_FALLBACK_FLAG=1 2022-03-20T15:13:48.927 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,270][ceph_volume.process][INFO ] stdout DM_UDEV_PRIMARY_SOURCE_FLAG=1 2022-03-20T15:13:48.928 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,270][ceph_volume.process][INFO ] stdout DM_UDEV_RULES=1 2022-03-20T15:13:48.928 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,270][ceph_volume.process][INFO ] stdout DM_UDEV_RULES_VSN=2 2022-03-20T15:13:48.928 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,270][ceph_volume.process][INFO ] stdout DM_UUID=LVM-ce31WZSvS0kvk6a73YxzJD4NdAMzmzV71SYBD8Ty1GaH36y2Kv1lDEBNUTJpC8Kw 2022-03-20T15:13:48.928 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,270][ceph_volume.process][INFO ] stdout DM_VG_NAME=vg_nvme 2022-03-20T15:13:48.929 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,270][ceph_volume.process][INFO ] stdout MAJOR=253 2022-03-20T15:13:48.929 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,270][ceph_volume.process][INFO ] stdout MINOR=2 2022-03-20T15:13:48.929 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,270][ceph_volume.process][INFO ] stdout SUBSYSTEM=block 2022-03-20T15:13:48.929 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,271][ceph_volume.process][INFO ] stdout TAGS=:systemd: 2022-03-20T15:13:48.930 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,271][ceph_volume.process][INFO ] stdout USEC_INITIALIZED=225190740 2022-03-20T15:13:48.930 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,271][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-authtool --gen-print-key 2022-03-20T15:13:48.930 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,287][ceph_volume.process][INFO ] stdout AQCcRDdi0IcNERAATrotPEVMD5pnNBpMm46YCw== 2022-03-20T15:13:48.930 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,288][ceph_volume.process][INFO ] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 89c05dd8-66c5-4120-bcd9-36717b1f4933 2022-03-20T15:13:48.931 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,435][ceph_volume.process][INFO ] stderr unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-20T15:13:48.931 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,435][ceph_volume.process][INFO ] stderr 2022-03-20T15:13:32.431+0000 7fd4db112700 -1 failed for service _ceph-mon._tcp 2022-03-20T15:13:48.931 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,435][ceph_volume.process][INFO ] stderr 2022-03-20T15:13:32.431+0000 7fd4db112700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-20T15:13:48.932 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,436][ceph_volume.process][INFO ] stderr [errno 2] RADOS object not found (error connecting to the cluster) 2022-03-20T15:13:48.932 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,436][ceph_volume.devices.raw.prepare][ERROR ] raw prepare was unable to complete 2022-03-20T15:13:48.932 INFO:tasks.rook.operator.smithi006.stdout:Traceback (most recent call last): 2022-03-20T15:13:48.932 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-03-20T15:13:48.933 INFO:tasks.rook.operator.smithi006.stdout: self.prepare() 2022-03-20T15:13:48.933 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-03-20T15:13:48.933 INFO:tasks.rook.operator.smithi006.stdout: return func(*a, **kw) 2022-03-20T15:13:48.933 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-03-20T15:13:48.933 INFO:tasks.rook.operator.smithi006.stdout: osd_fsid, json.dumps(secrets)) 2022-03-20T15:13:48.934 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-03-20T15:13:48.934 INFO:tasks.rook.operator.smithi006.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-03-20T15:13:48.934 INFO:tasks.rook.operator.smithi006.stdout:RuntimeError: Unable to create a new OSD id 2022-03-20T15:13:48.934 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,437][ceph_volume.devices.raw.prepare][INFO ] will rollback OSD ID creation 2022-03-20T15:13:48.934 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:32,437][ceph_volume][ERROR ] exception caught by decorator 2022-03-20T15:13:48.935 INFO:tasks.rook.operator.smithi006.stdout:Traceback (most recent call last): 2022-03-20T15:13:48.935 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2022-03-20T15:13:48.935 INFO:tasks.rook.operator.smithi006.stdout: return f(*a, **kw) 2022-03-20T15:13:48.935 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2022-03-20T15:13:48.936 INFO:tasks.rook.operator.smithi006.stdout: terminal.dispatch(self.mapper, subcommand_args) 2022-03-20T15:13:48.936 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-20T15:13:48.936 INFO:tasks.rook.operator.smithi006.stdout: instance.main() 2022-03-20T15:13:48.936 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main 2022-03-20T15:13:48.936 INFO:tasks.rook.operator.smithi006.stdout: terminal.dispatch(self.mapper, self.argv) 2022-03-20T15:13:48.937 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-20T15:13:48.937 INFO:tasks.rook.operator.smithi006.stdout: instance.main() 2022-03-20T15:13:48.937 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main 2022-03-20T15:13:48.937 INFO:tasks.rook.operator.smithi006.stdout: self.safe_prepare(self.args) 2022-03-20T15:13:48.938 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-03-20T15:13:48.938 INFO:tasks.rook.operator.smithi006.stdout: self.prepare() 2022-03-20T15:13:48.938 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-03-20T15:13:48.938 INFO:tasks.rook.operator.smithi006.stdout: return func(*a, **kw) 2022-03-20T15:13:48.939 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-03-20T15:13:48.939 INFO:tasks.rook.operator.smithi006.stdout: osd_fsid, json.dumps(secrets)) 2022-03-20T15:13:48.939 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-03-20T15:13:48.939 INFO:tasks.rook.operator.smithi006.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-03-20T15:13:48.939 INFO:tasks.rook.operator.smithi006.stdout:RuntimeError: Unable to create a new OSD id 2022-03-20T15:13:48.940 INFO:tasks.rook.operator.smithi006.stdout:: exit status 1} 2022-03-20T15:13:48.940 INFO:tasks.rook.operator.smithi006.stdout:failed to provision OSD(s) on PVC smithi123-lv-3-data-0crj7j. &{OSDs:[] Status:failed PvcBackedOSD:true Message:failed to configure devices: failed to initialize devices on PVC: failed to run ceph-volume. Running command: /usr/bin/ceph-authtool --gen-print-key 2022-03-20T15:13:48.940 INFO:tasks.rook.operator.smithi006.stdout:Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4c3a0862-b08a-4b00-b105-3c1e58b3fe99 2022-03-20T15:13:48.940 INFO:tasks.rook.operator.smithi006.stdout: stderr: unable to get monitor info from DNS SRV with service name: 2022-03-20T15:13:34.391+0000 7ffb6611b700 -1 failed for service _ceph-mon._tcp 2022-03-20T15:13:48.941 INFO:tasks.rook.operator.smithi006.stdout: stderr: ceph-mon 2022-03-20T15:13:48.941 INFO:tasks.rook.operator.smithi006.stdout: stderr: 2022-03-20T15:13:34.391+0000 7ffb6611b700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-20T15:13:48.941 INFO:tasks.rook.operator.smithi006.stdout: stderr: [errno 2] RADOS object not found (error connecting to the cluster) 2022-03-20T15:13:48.941 INFO:tasks.rook.operator.smithi006.stdout:Traceback (most recent call last): 2022-03-20T15:13:48.942 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/sbin/ceph-volume", line 11, in 2022-03-20T15:13:48.942 INFO:tasks.rook.operator.smithi006.stdout: load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2022-03-20T15:13:48.942 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2022-03-20T15:13:48.943 INFO:tasks.rook.operator.smithi006.stdout: self.main(self.argv) 2022-03-20T15:13:48.943 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2022-03-20T15:13:48.943 INFO:tasks.rook.operator.smithi006.stdout: return f(*a, **kw) 2022-03-20T15:13:48.943 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2022-03-20T15:13:48.944 INFO:tasks.rook.operator.smithi006.stdout: terminal.dispatch(self.mapper, subcommand_args) 2022-03-20T15:13:48.944 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-20T15:13:48.944 INFO:tasks.rook.operator.smithi006.stdout: instance.main() 2022-03-20T15:13:48.945 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main 2022-03-20T15:13:48.945 INFO:tasks.rook.operator.smithi006.stdout: terminal.dispatch(self.mapper, self.argv) 2022-03-20T15:13:48.945 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-20T15:13:48.945 INFO:tasks.rook.operator.smithi006.stdout: instance.main() 2022-03-20T15:13:48.946 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main 2022-03-20T15:13:48.946 INFO:tasks.rook.operator.smithi006.stdout: self.safe_prepare(self.args) 2022-03-20T15:13:48.946 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-03-20T15:13:48.946 INFO:tasks.rook.operator.smithi006.stdout: self.prepare() 2022-03-20T15:13:48.947 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-03-20T15:13:48.947 INFO:tasks.rook.operator.smithi006.stdout: return func(*a, **kw) 2022-03-20T15:13:48.947 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-03-20T15:13:48.948 INFO:tasks.rook.operator.smithi006.stdout: osd_fsid, json.dumps(secrets)) 2022-03-20T15:13:48.948 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-03-20T15:13:48.948 INFO:tasks.rook.operator.smithi006.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-03-20T15:13:48.948 INFO:tasks.rook.operator.smithi006.stdout:RuntimeError: Unable to create a new OSD id. debug logs below: 2022-03-20T15:13:48.949 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,014][ceph_volume.main][INFO ] Running command: ceph-volume --log-path /var/log/ceph/smithi123-lv-3-data-0crj7j raw prepare --bluestore --data /dev/mapper/vg_nvme-lv_3 2022-03-20T15:13:48.949 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,014][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk -plno KNAME,NAME,TYPE 2022-03-20T15:13:48.949 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,024][ceph_volume.process][INFO ] stdout /dev/loop0 /dev/loop0 loop 2022-03-20T15:13:48.949 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,024][ceph_volume.process][INFO ] stdout /dev/loop1 /dev/loop1 loop 2022-03-20T15:13:48.949 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,024][ceph_volume.process][INFO ] stdout /dev/loop2 /dev/loop2 loop 2022-03-20T15:13:48.950 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,024][ceph_volume.process][INFO ] stdout /dev/loop3 /dev/loop3 loop 2022-03-20T15:13:48.950 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,024][ceph_volume.process][INFO ] stdout /dev/sda /dev/sda disk 2022-03-20T15:13:48.950 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,024][ceph_volume.process][INFO ] stdout /dev/sda1 /dev/sda1 part 2022-03-20T15:13:48.950 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,024][ceph_volume.process][INFO ] stdout /dev/dm-0 /dev/mapper/vg_nvme-lv_1 lvm 2022-03-20T15:13:48.951 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,024][ceph_volume.process][INFO ] stdout /dev/dm-1 /dev/mapper/vg_nvme-lv_2 lvm 2022-03-20T15:13:48.951 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,024][ceph_volume.process][INFO ] stdout /dev/dm-2 /dev/mapper/vg_nvme-lv_3 lvm 2022-03-20T15:13:48.952 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,025][ceph_volume.process][INFO ] stdout /dev/dm-3 /dev/mapper/vg_nvme-lv_4 lvm 2022-03-20T15:13:48.952 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,025][ceph_volume.process][INFO ] stdout /dev/dm-4 /dev/mapper/vg_nvme-lv_5 lvm 2022-03-20T15:13:48.952 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,025][ceph_volume.process][INFO ] stdout /dev/nvme0n1 /dev/nvme0n1 disk 2022-03-20T15:13:48.952 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,025][ceph_volume.process][INFO ] stdout /dev/nvme1n1 /dev/nvme1n1 disk 2022-03-20T15:13:48.953 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,025][ceph_volume.process][INFO ] stdout /dev/nvme2n1 /dev/nvme2n1 disk 2022-03-20T15:13:48.953 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,025][ceph_volume.process][INFO ] stdout /dev/nvme3n1 /dev/nvme3n1 disk 2022-03-20T15:13:48.953 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,025][ceph_volume.process][INFO ] stdout /dev/nvme4n1 /dev/nvme4n1 disk 2022-03-20T15:13:48.953 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,035][ceph_volume.util.system][WARNING] Executable lvs not found on the host, will return lvs as-is 2022-03-20T15:13:48.953 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,035][ceph_volume.process][INFO ] Running command: lvs --noheadings --readonly --separator=";" -a --units=b --nosuffix -S lv_path=/dev/mapper/vg_nvme-lv_3 -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size 2022-03-20T15:13:48.954 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,243][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk --nodeps -P -o NAME,KNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL /dev/mapper/vg_nvme-lv_3 2022-03-20T15:13:48.954 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,248][ceph_volume.process][INFO ] stdout NAME="vg_nvme-lv_3" KNAME="dm-2" MAJ:MIN="253:2" FSTYPE="" MOUNTPOINT="" LABEL="" UUID="" RO="0" RM="0" MODEL="" SIZE="89.4G" STATE="running" OWNER="root" GROUP="disk" MODE="brw-rw----" ALIGNMENT="0" PHY-SEC="512" LOG-SEC="512" ROTA="0" SCHED="" TYPE="lvm" DISC-ALN="0" DISC-GRAN="512B" DISC-MAX="2T" DISC-ZERO="0" PKNAME="" PARTLABEL="" 2022-03-20T15:13:48.954 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,248][ceph_volume.process][INFO ] Running command: /usr/sbin/blkid -c /dev/null -p /dev/mapper/vg_nvme-lv_3 2022-03-20T15:13:48.954 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,251][ceph_volume.util.disk][INFO ] opening device /dev/mapper/vg_nvme-lv_3 to check for BlueStore label 2022-03-20T15:13:48.955 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,252][ceph_volume.util.disk][INFO ] opening device /dev/mapper/vg_nvme-lv_3 to check for BlueStore label 2022-03-20T15:13:48.955 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,252][ceph_volume.process][INFO ] Running command: /usr/sbin/udevadm info --query=property /dev/mapper/vg_nvme-lv_3 2022-03-20T15:13:48.955 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,259][ceph_volume.process][INFO ] stdout DEVLINKS=/dev/mapper/vg_nvme-lv_3 /dev/disk/by-id/dm-name-vg_nvme-lv_3 /dev/disk/by-id/dm-uuid-LVM-PZaq4q5Lcwhjnsid2JGepycLeSgZYtJ62wOvEg7V5hlog53S7OZJodacRc6Mxd1T /dev/vg_nvme/lv_3 2022-03-20T15:13:48.955 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,259][ceph_volume.process][INFO ] stdout DEVNAME=/dev/dm-2 2022-03-20T15:13:48.956 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,259][ceph_volume.process][INFO ] stdout DEVPATH=/devices/virtual/block/dm-2 2022-03-20T15:13:48.956 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,260][ceph_volume.process][INFO ] stdout DEVTYPE=disk 2022-03-20T15:13:48.956 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,260][ceph_volume.process][INFO ] stdout DM_LV_NAME=lv_3 2022-03-20T15:13:48.956 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,260][ceph_volume.process][INFO ] stdout DM_NAME=vg_nvme-lv_3 2022-03-20T15:13:48.957 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,260][ceph_volume.process][INFO ] stdout DM_SUSPENDED=0 2022-03-20T15:13:48.957 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,260][ceph_volume.process][INFO ] stdout DM_UDEV_DISABLE_LIBRARY_FALLBACK_FLAG=1 2022-03-20T15:13:48.957 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,260][ceph_volume.process][INFO ] stdout DM_UDEV_PRIMARY_SOURCE_FLAG=1 2022-03-20T15:13:48.957 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,260][ceph_volume.process][INFO ] stdout DM_UDEV_RULES=1 2022-03-20T15:13:48.958 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,260][ceph_volume.process][INFO ] stdout DM_UDEV_RULES_VSN=2 2022-03-20T15:13:48.958 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,260][ceph_volume.process][INFO ] stdout DM_UUID=LVM-PZaq4q5Lcwhjnsid2JGepycLeSgZYtJ62wOvEg7V5hlog53S7OZJodacRc6Mxd1T 2022-03-20T15:13:48.958 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,260][ceph_volume.process][INFO ] stdout DM_VG_NAME=vg_nvme 2022-03-20T15:13:48.958 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,260][ceph_volume.process][INFO ] stdout MAJOR=253 2022-03-20T15:13:48.959 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,260][ceph_volume.process][INFO ] stdout MINOR=2 2022-03-20T15:13:48.959 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,260][ceph_volume.process][INFO ] stdout SUBSYSTEM=block 2022-03-20T15:13:48.959 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,260][ceph_volume.process][INFO ] stdout TAGS=:systemd: 2022-03-20T15:13:48.959 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,260][ceph_volume.process][INFO ] stdout USEC_INITIALIZED=222914530 2022-03-20T15:13:48.960 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,261][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-authtool --gen-print-key 2022-03-20T15:13:48.960 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,281][ceph_volume.process][INFO ] stdout AQCeRDdijHCvEBAA+O3GAhABVcCaZ8CQ2t7AEQ== 2022-03-20T15:13:48.960 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,281][ceph_volume.process][INFO ] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4c3a0862-b08a-4b00-b105-3c1e58b3fe99 2022-03-20T15:13:48.960 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,397][ceph_volume.process][INFO ] stderr unable to get monitor info from DNS SRV with service name: 2022-03-20T15:13:34.391+0000 7ffb6611b700 -1 failed for service _ceph-mon._tcp 2022-03-20T15:13:48.960 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,397][ceph_volume.process][INFO ] stderr ceph-mon 2022-03-20T15:13:48.961 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,397][ceph_volume.process][INFO ] stderr 2022-03-20T15:13:34.391+0000 7ffb6611b700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-20T15:13:48.961 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,397][ceph_volume.process][INFO ] stderr [errno 2] RADOS object not found (error connecting to the cluster) 2022-03-20T15:13:48.961 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,397][ceph_volume.devices.raw.prepare][ERROR ] raw prepare was unable to complete 2022-03-20T15:13:48.961 INFO:tasks.rook.operator.smithi006.stdout:Traceback (most recent call last): 2022-03-20T15:13:48.962 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-03-20T15:13:48.962 INFO:tasks.rook.operator.smithi006.stdout: self.prepare() 2022-03-20T15:13:48.962 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-03-20T15:13:48.962 INFO:tasks.rook.operator.smithi006.stdout: return func(*a, **kw) 2022-03-20T15:13:48.963 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-03-20T15:13:48.963 INFO:tasks.rook.operator.smithi006.stdout: osd_fsid, json.dumps(secrets)) 2022-03-20T15:13:48.963 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-03-20T15:13:48.964 INFO:tasks.rook.operator.smithi006.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-03-20T15:13:48.964 INFO:tasks.rook.operator.smithi006.stdout:RuntimeError: Unable to create a new OSD id 2022-03-20T15:13:48.964 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,398][ceph_volume.devices.raw.prepare][INFO ] will rollback OSD ID creation 2022-03-20T15:13:48.964 INFO:tasks.rook.operator.smithi006.stdout:[2022-03-20 15:13:34,398][ceph_volume][ERROR ] exception caught by decorator 2022-03-20T15:13:48.965 INFO:tasks.rook.operator.smithi006.stdout:Traceback (most recent call last): 2022-03-20T15:13:48.965 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2022-03-20T15:13:48.965 INFO:tasks.rook.operator.smithi006.stdout: return f(*a, **kw) 2022-03-20T15:13:48.965 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2022-03-20T15:13:48.966 INFO:tasks.rook.operator.smithi006.stdout: terminal.dispatch(self.mapper, subcommand_args) 2022-03-20T15:13:48.966 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-20T15:13:48.966 INFO:tasks.rook.operator.smithi006.stdout: instance.main() 2022-03-20T15:13:48.967 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main 2022-03-20T15:13:48.967 INFO:tasks.rook.operator.smithi006.stdout: terminal.dispatch(self.mapper, self.argv) 2022-03-20T15:13:48.967 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-20T15:13:48.967 INFO:tasks.rook.operator.smithi006.stdout: instance.main() 2022-03-20T15:13:48.968 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main 2022-03-20T15:13:48.968 INFO:tasks.rook.operator.smithi006.stdout: self.safe_prepare(self.args) 2022-03-20T15:13:48.968 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-03-20T15:13:48.968 INFO:tasks.rook.operator.smithi006.stdout: self.prepare() 2022-03-20T15:13:48.969 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-03-20T15:13:48.969 INFO:tasks.rook.operator.smithi006.stdout: return func(*a, **kw) 2022-03-20T15:13:48.969 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-03-20T15:13:48.969 INFO:tasks.rook.operator.smithi006.stdout: osd_fsid, json.dumps(secrets)) 2022-03-20T15:13:48.969 INFO:tasks.rook.operator.smithi006.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-03-20T15:13:48.970 INFO:tasks.rook.operator.smithi006.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-03-20T15:13:48.970 INFO:tasks.rook.operator.smithi006.stdout:RuntimeError: Unable to create a new OSD id 2022-03-20T15:13:48.970 INFO:tasks.rook.operator.smithi006.stdout:: exit status 1} 2022-03-20T15:13:48.970 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:48.902641 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-20T15:13:49.757 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:49.755548 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:13:50.136 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:13:50.135414 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e... 2022-03-20T15:13:58.325 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd stat -f json 2022-03-20T15:13:59.581 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:13:59.606 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":15,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1647789222,"num_remapped_pgs":0} 2022-03-20T15:13:59.607 INFO:tasks.rook: have 0/12 OSDs 2022-03-20T15:14:09.609 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd stat -f json 2022-03-20T15:14:11.277 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:14:11.306 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":16,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1647789222,"num_remapped_pgs":0} 2022-03-20T15:14:11.306 INFO:tasks.rook: have 0/12 OSDs 2022-03-20T15:14:15.545 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:15.542102 I | ceph-spec: detected ceph image version: "17.0.0-11187 quincy" 2022-03-20T15:14:15.546 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:15.542142 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-20T15:14:15.775 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:15.772609 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:14:15.953 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:15.950849 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T15:14:15.960 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:15.951235 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T15:14:16.441 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:16.438823 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11187 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e" 2022-03-20T15:14:18.130 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:18.122470 I | op-mon: start running mons 2022-03-20T15:14:18.352 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:18.349629 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:14:18.595 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:18.592495 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.13.112:6789","10.99.78.235:6789","10.107.211.141:6789"]}] data:c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 mapping:{"node":{"a":{"Name":"smithi045","Hostname":"smithi045","Address":"172.21.15.45"},"b":{"Name":"smithi123","Hostname":"smithi123","Address":"172.21.15.123"},"c":{"Name":"smithi006","Hostname":"smithi006","Address":"172.21.15.6"}}} maxMonId:2] 2022-03-20T15:14:19.391 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:19.388454 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T15:14:19.396 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:19.388846 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T15:14:20.805 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:20.802827 I | op-mon: targeting the mon count 3 2022-03-20T15:14:20.847 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:20.845211 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T15:14:21.271 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:21.268612 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T15:14:21.272 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:21.268652 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T15:14:21.308 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd stat -f json 2022-03-20T15:14:21.632 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:21.629677 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T15:14:21.633 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:21.629706 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T15:14:21.881 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:14:21.903 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"num_osds":12,"num_up_osds":5,"osd_up_since":1647789261,"num_in_osds":12,"osd_in_since":1647789222,"num_remapped_pgs":0} 2022-03-20T15:14:21.904 INFO:tasks.rook: have 5/12 OSDs 2022-03-20T15:14:21.970 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:21.967330 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T15:14:21.970 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:21.967361 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-20T15:14:22.692 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:22.688837 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-20T15:14:22.692 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:22.688878 I | op-config: deleting "log file" option from the mon configuration database 2022-03-20T15:14:24.013 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:24.009789 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-20T15:14:24.013 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:24.009842 I | op-mon: checking for basic quorum with existing mons 2022-03-20T15:14:25.369 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:25.365929 I | op-mon: mon "b" ip is 10.97.13.112 2022-03-20T15:14:25.861 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:25.857991 I | op-mon: mon "c" ip is 10.99.78.235 2022-03-20T15:14:26.239 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:26.235934 I | op-mon: mon "a" ip is 10.107.211.141 2022-03-20T15:14:26.352 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:26.349558 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.99.78.235:6789","10.107.211.141:6789","10.97.13.112:6789"]}] data:c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 mapping:{"node":{"a":{"Name":"smithi045","Hostname":"smithi045","Address":"172.21.15.45"},"b":{"Name":"smithi123","Hostname":"smithi123","Address":"172.21.15.123"},"c":{"Name":"smithi006","Hostname":"smithi006","Address":"172.21.15.6"}}} maxMonId:2] 2022-03-20T15:14:26.372 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:26.369753 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T15:14:26.373 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:26.369882 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T15:14:26.796 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:26.793445 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-20T15:14:26.804 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:26.801669 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-20T15:14:26.805 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:26.801689 I | op-mon: waiting for mon quorum with [b c a] 2022-03-20T15:14:27.416 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:27.413266 I | op-mon: mons running: [b c a] 2022-03-20T15:14:27.902 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:27.899544 I | op-mon: Monitors in quorum: [a b c] 2022-03-20T15:14:27.956 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:27.952946 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-20T15:14:27.968 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:27.965623 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-20T15:14:27.968 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:27.965659 I | op-mon: waiting for mon quorum with [b c a] 2022-03-20T15:14:28.038 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:28.035802 I | op-mon: mons running: [b c a] 2022-03-20T15:14:28.476 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:28.473538 I | op-mon: Monitors in quorum: [a b c] 2022-03-20T15:14:28.485 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:28.482283 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-20T15:14:28.499 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:28.496734 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-20T15:14:28.500 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:28.496768 I | op-mon: waiting for mon quorum with [b c a] 2022-03-20T15:14:28.607 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:28.604495 I | op-mon: mons running: [b c a] 2022-03-20T15:14:29.059 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:29.056167 I | op-mon: Monitors in quorum: [a b c] 2022-03-20T15:14:29.059 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:29.056200 I | op-mon: mons created: 3 2022-03-20T15:14:29.493 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:29.490065 I | op-mon: waiting for mon quorum with [b c a] 2022-03-20T15:14:29.527 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:29.525010 I | op-mon: mons running: [b c a] 2022-03-20T15:14:29.950 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:29.946735 I | op-mon: Monitors in quorum: [a b c] 2022-03-20T15:14:29.950 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:29.946764 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-20T15:14:30.445 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:30.441545 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-20T15:14:30.857 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:30.853754 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-20T15:14:31.355 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:31.351491 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-20T15:14:31.875 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:31.871672 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-20T15:14:31.875 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:31.871701 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-20T15:14:31.905 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd stat -f json 2022-03-20T15:14:32.310 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:32.307514 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-20T15:14:32.311 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:32.307533 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-20T15:14:32.311 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:32.307538 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-20T15:14:32.504 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:14:32.525 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":29,"num_osds":12,"num_up_osds":10,"osd_up_since":1647789272,"num_in_osds":12,"osd_in_since":1647789222,"num_remapped_pgs":1} 2022-03-20T15:14:32.526 INFO:tasks.rook: have 10/12 OSDs 2022-03-20T15:14:32.752 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:32.748757 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-20T15:14:32.788 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:32.785880 I | op-mgr: start running mgr 2022-03-20T15:14:32.789 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:32.785898 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-20T15:14:33.285 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:33.282177 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-20T15:14:33.295 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:33.292443 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-20T15:14:33.298 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:33.295693 I | op-mgr: setting services to point to mgr "a" 2022-03-20T15:14:33.835 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:33.832319 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-20T15:14:33.836 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:33.832404 I | op-mgr: successful modules: balancer 2022-03-20T15:14:33.848 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:33.845009 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-20T15:14:33.849 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:33.845049 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-20T15:14:33.913 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:33.910164 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-20T15:14:34.040 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.034757 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi006-lv-1" 2022-03-20T15:14:34.041 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.034814 I | op-osd: OSD PVC "smithi006-lv-1-data-0cmfzx" already exists 2022-03-20T15:14:34.041 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.034838 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi006-lv-2" 2022-03-20T15:14:34.041 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.034864 I | op-osd: OSD PVC "smithi006-lv-2-data-0hwbjq" already exists 2022-03-20T15:14:34.041 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.034882 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi006-lv-3" 2022-03-20T15:14:34.042 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.034919 I | op-osd: OSD PVC "smithi006-lv-3-data-07dd5g" already exists 2022-03-20T15:14:34.042 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.034940 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi006-lv-4" 2022-03-20T15:14:34.042 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.034966 I | op-osd: OSD PVC "smithi006-lv-4-data-0qbdz5" already exists 2022-03-20T15:14:34.042 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.034988 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi045-lv-1" 2022-03-20T15:14:34.043 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.035017 I | op-osd: OSD PVC "smithi045-lv-1-data-0rdlmk" already exists 2022-03-20T15:14:34.043 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.035044 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi045-lv-2" 2022-03-20T15:14:34.043 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.035064 I | op-osd: OSD PVC "smithi045-lv-2-data-0lwhr2" already exists 2022-03-20T15:14:34.043 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.035077 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi045-lv-3" 2022-03-20T15:14:34.044 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.035097 I | op-osd: OSD PVC "smithi045-lv-3-data-0jqzrc" already exists 2022-03-20T15:14:34.044 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.035111 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi045-lv-4" 2022-03-20T15:14:34.044 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.035131 I | op-osd: OSD PVC "smithi045-lv-4-data-026444" already exists 2022-03-20T15:14:34.044 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.035145 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi123-lv-1" 2022-03-20T15:14:34.045 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.035166 I | op-osd: OSD PVC "smithi123-lv-1-data-0vz4f8" already exists 2022-03-20T15:14:34.045 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.035189 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi123-lv-2" 2022-03-20T15:14:34.045 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.035213 I | op-osd: OSD PVC "smithi123-lv-2-data-0l58zl" already exists 2022-03-20T15:14:34.045 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.035241 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi123-lv-3" 2022-03-20T15:14:34.046 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.035262 I | op-osd: OSD PVC "smithi123-lv-3-data-0crj7j" already exists 2022-03-20T15:14:34.046 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.035277 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi123-lv-4" 2022-03-20T15:14:34.046 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.035296 I | op-osd: OSD PVC "smithi123-lv-4-data-0vfnsc" already exists 2022-03-20T15:14:34.069 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.064848 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-1-data-0cmfzx" 2022-03-20T15:14:34.069 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.064894 I | op-osd: skipping OSD prepare job creation for PVC "smithi006-lv-1-data-0cmfzx" because OSD daemon using the PVC already exists 2022-03-20T15:14:34.070 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.064904 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-2-data-0hwbjq" 2022-03-20T15:14:34.070 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.064913 I | op-osd: skipping OSD prepare job creation for PVC "smithi006-lv-2-data-0hwbjq" because OSD daemon using the PVC already exists 2022-03-20T15:14:34.070 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.064921 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-3-data-07dd5g" 2022-03-20T15:14:34.070 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.064936 I | op-osd: skipping OSD prepare job creation for PVC "smithi006-lv-3-data-07dd5g" because OSD daemon using the PVC already exists 2022-03-20T15:14:34.071 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.064948 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-4-data-0qbdz5" 2022-03-20T15:14:34.071 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.064956 I | op-osd: skipping OSD prepare job creation for PVC "smithi006-lv-4-data-0qbdz5" because OSD daemon using the PVC already exists 2022-03-20T15:14:34.071 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.064964 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-1-data-0rdlmk" 2022-03-20T15:14:34.071 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.064972 I | op-osd: skipping OSD prepare job creation for PVC "smithi045-lv-1-data-0rdlmk" because OSD daemon using the PVC already exists 2022-03-20T15:14:34.072 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.064990 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-2-data-0lwhr2" 2022-03-20T15:14:34.072 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.064999 I | op-osd: skipping OSD prepare job creation for PVC "smithi045-lv-2-data-0lwhr2" because OSD daemon using the PVC already exists 2022-03-20T15:14:34.072 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.065008 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-3-data-0jqzrc" 2022-03-20T15:14:34.458 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.454916 I | op-k8sutil: Removing previous job rook-ceph-osd-prepare-smithi045-lv-3-data-0jqzrc to start a new one 2022-03-20T15:14:34.515 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:34.511984 I | op-k8sutil: batch job rook-ceph-osd-prepare-smithi045-lv-3-data-0jqzrc still exists 2022-03-20T15:14:35.071 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:35.067688 I | op-mgr: successful modules: prometheus 2022-03-20T15:14:35.071 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:35.067941 I | op-mgr: successful modules: dashboard 2022-03-20T15:14:35.075 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:35.071773 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-20T15:14:36.813 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:36.809965 I | op-mgr: successful modules: orchestrator modules 2022-03-20T15:14:37.556 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:37.552874 I | op-k8sutil: batch job rook-ceph-osd-prepare-smithi045-lv-3-data-0jqzrc deleted 2022-03-20T15:14:37.564 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:37.560798 I | op-osd: started OSD provisioning job for PVC "smithi045-lv-3-data-0jqzrc" 2022-03-20T15:14:37.564 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:37.560819 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-4-data-026444" 2022-03-20T15:14:37.565 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:37.560824 I | op-osd: skipping OSD prepare job creation for PVC "smithi045-lv-4-data-026444" because OSD daemon using the PVC already exists 2022-03-20T15:14:37.565 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:37.560828 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-1-data-0vz4f8" 2022-03-20T15:14:37.565 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:37.560833 I | op-osd: skipping OSD prepare job creation for PVC "smithi123-lv-1-data-0vz4f8" because OSD daemon using the PVC already exists 2022-03-20T15:14:37.565 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:37.560837 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-2-data-0l58zl" 2022-03-20T15:14:37.566 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:37.560842 I | op-osd: skipping OSD prepare job creation for PVC "smithi123-lv-2-data-0l58zl" because OSD daemon using the PVC already exists 2022-03-20T15:14:37.566 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:37.560846 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-3-data-0crj7j" 2022-03-20T15:14:37.710 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:37.706566 I | op-k8sutil: Removing previous job rook-ceph-osd-prepare-smithi123-lv-3-data-0crj7j to start a new one 2022-03-20T15:14:37.761 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:37.757662 I | op-k8sutil: batch job rook-ceph-osd-prepare-smithi123-lv-3-data-0crj7j still exists 2022-03-20T15:14:40.765 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:40.762145 I | op-k8sutil: batch job rook-ceph-osd-prepare-smithi123-lv-3-data-0crj7j deleted 2022-03-20T15:14:40.773 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:40.769549 I | op-osd: started OSD provisioning job for PVC "smithi123-lv-3-data-0crj7j" 2022-03-20T15:14:40.773 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:40.769579 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-4-data-0vfnsc" 2022-03-20T15:14:40.773 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:40.769592 I | op-osd: skipping OSD prepare job creation for PVC "smithi123-lv-4-data-0vfnsc" because OSD daemon using the PVC already exists 2022-03-20T15:14:40.774 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:40.769605 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-20T15:14:40.774 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:40.769619 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-20T15:14:40.824 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:40.819560 I | op-osd: OSD orchestration status for PVC smithi045-lv-3-data-0jqzrc is "starting" 2022-03-20T15:14:40.824 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:40.819607 I | op-osd: OSD orchestration status for PVC smithi123-lv-3-data-0crj7j is "starting" 2022-03-20T15:14:40.927 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:40.922860 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:14:40.996 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:40.992879 I | op-osd: updating OSD 0 on PVC "smithi045-lv-1-data-0rdlmk" 2022-03-20T15:14:40.997 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:40.992920 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-1-data-0rdlmk" 2022-03-20T15:14:41.198 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:41.195036 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:14:41.204 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:41.200729 I | op-osd: updating OSD 1 on PVC "smithi006-lv-3-data-07dd5g" 2022-03-20T15:14:41.204 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:41.200755 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-3-data-07dd5g" 2022-03-20T15:14:41.385 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:41.382333 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:14:41.392 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:41.388954 I | op-osd: updating OSD 2 on PVC "smithi123-lv-1-data-0vz4f8" 2022-03-20T15:14:41.392 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:41.388989 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-1-data-0vz4f8" 2022-03-20T15:14:41.579 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:41.575856 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:14:41.589 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:41.586358 I | op-osd: updating OSD 3 on PVC "smithi045-lv-2-data-0lwhr2" 2022-03-20T15:14:41.589 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:41.586376 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-2-data-0lwhr2" 2022-03-20T15:14:42.238 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:42.234960 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:14:42.332 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:42.328972 I | op-osd: updating OSD 4 on PVC "smithi006-lv-2-data-0hwbjq" 2022-03-20T15:14:42.332 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:42.328994 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-2-data-0hwbjq" 2022-03-20T15:14:42.527 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd stat -f json 2022-03-20T15:14:43.044 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:43.041150 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:14:43.141 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:43.134420 I | op-osd: updating OSD 5 on PVC "smithi006-lv-4-data-0qbdz5" 2022-03-20T15:14:43.142 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:43.134449 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-4-data-0qbdz5" 2022-03-20T15:14:43.180 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:14:43.199 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":33,"num_osds":12,"num_up_osds":10,"osd_up_since":1647789272,"num_in_osds":12,"osd_in_since":1647789222,"num_remapped_pgs":0} 2022-03-20T15:14:43.199 INFO:tasks.rook: have 10/12 OSDs 2022-03-20T15:14:43.837 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:43.834122 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:14:43.934 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:43.930883 I | op-osd: updating OSD 6 on PVC "smithi006-lv-1-data-0cmfzx" 2022-03-20T15:14:43.934 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:43.930917 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-1-data-0cmfzx" 2022-03-20T15:14:44.642 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:44.639509 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:14:44.740 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:44.737543 I | op-osd: updating OSD 7 on PVC "smithi045-lv-4-data-026444" 2022-03-20T15:14:44.741 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:44.737580 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-4-data-026444" 2022-03-20T15:14:45.440 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:45.437450 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:14:45.549 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:45.543165 I | op-osd: updating OSD 8 on PVC "smithi123-lv-4-data-0vfnsc" 2022-03-20T15:14:45.550 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:45.543204 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-4-data-0vfnsc" 2022-03-20T15:14:46.138 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:46.135263 I | op-osd: OSD orchestration status for node smithi045-lv-3-data-0jqzrc is "orchestrating" 2022-03-20T15:14:46.139 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:46.135568 I | op-osd: OSD orchestration status for PVC smithi045-lv-3-data-0jqzrc is "orchestrating" 2022-03-20T15:14:46.239 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:46.236305 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:14:46.334 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:46.330450 I | op-osd: updating OSD 9 on PVC "smithi123-lv-2-data-0l58zl" 2022-03-20T15:14:46.334 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:46.330477 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-2-data-0l58zl" 2022-03-20T15:14:47.002 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:46.999536 I | op-osd: OSD orchestration status for PVC smithi045-lv-3-data-0jqzrc is "completed" 2022-03-20T15:14:47.003 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:46.999557 I | op-osd: creating OSD 11 on PVC "smithi045-lv-3-data-0jqzrc" 2022-03-20T15:14:47.003 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:46.999565 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-3-data-0jqzrc" 2022-03-20T15:14:48.060 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:48.056930 I | op-osd: OSD orchestration status for node smithi123-lv-3-data-0crj7j is "orchestrating" 2022-03-20T15:14:48.141 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:48.138191 I | op-osd: OSD orchestration status for PVC smithi123-lv-3-data-0crj7j is "orchestrating" 2022-03-20T15:14:49.262 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:49.259222 I | op-osd: OSD orchestration status for PVC smithi123-lv-3-data-0crj7j is "completed" 2022-03-20T15:14:49.263 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:49.259263 I | op-osd: creating OSD 10 on PVC "smithi123-lv-3-data-0crj7j" 2022-03-20T15:14:49.263 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:49.259282 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-3-data-0crj7j" 2022-03-20T15:14:50.249 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:50.245431 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-20T15:14:50.249 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:14:50.245448 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-20T15:14:53.200 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd stat -f json 2022-03-20T15:14:53.726 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:14:53.744 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":33,"num_osds":12,"num_up_osds":10,"osd_up_since":1647789272,"num_in_osds":12,"osd_in_since":1647789222,"num_remapped_pgs":0} 2022-03-20T15:14:53.745 INFO:tasks.rook: have 10/12 OSDs 2022-03-20T15:15:03.746 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd stat -f json 2022-03-20T15:15:04.343 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:15:04.374 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":33,"num_osds":12,"num_up_osds":10,"osd_up_since":1647789272,"num_in_osds":12,"osd_in_since":1647789222,"num_remapped_pgs":0} 2022-03-20T15:15:04.375 INFO:tasks.rook: have 10/12 OSDs 2022-03-20T15:15:14.376 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd stat -f json 2022-03-20T15:15:14.842 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:15:14.861 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":40,"num_osds":12,"num_up_osds":12,"osd_up_since":1647789310,"num_in_osds":12,"osd_in_since":1647789222,"num_remapped_pgs":0} 2022-03-20T15:15:14.862 INFO:tasks.rook:Distributing ceph config and client.admin keyring 2022-03-20T15:15:14.862 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- cat /etc/ceph/ceph.conf 2022-03-20T15:15:15.068 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2022-03-20T15:15:15.068 INFO:teuthology.orchestra.run.smithi006.stdout:mon_host = 10.99.78.235:6789,10.107.211.141:6789,10.97.13.112:6789 2022-03-20T15:15:15.069 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:15:15.069 INFO:teuthology.orchestra.run.smithi006.stdout:[client.admin] 2022-03-20T15:15:15.069 INFO:teuthology.orchestra.run.smithi006.stdout:keyring = /etc/ceph/keyring 2022-03-20T15:15:15.073 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- cat /etc/ceph/keyring 2022-03-20T15:15:15.251 INFO:teuthology.orchestra.run.smithi006.stdout:[client.admin] 2022-03-20T15:15:15.251 INFO:teuthology.orchestra.run.smithi006.stdout:key = AQCFQzdie4DPEBAAlwXCwCLIccPqM3kbDI0l2A== 2022-03-20T15:15:15.255 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph 2022-03-20T15:15:15.269 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /etc/ceph 2022-03-20T15:15:15.287 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /etc/ceph 2022-03-20T15:15:15.302 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2022-03-20T15:15:15.303 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/ceph.conf 2022-03-20T15:15:15.318 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2022-03-20T15:15:15.318 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/keyring 2022-03-20T15:15:15.375 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2022-03-20T15:15:15.375 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/ceph.conf 2022-03-20T15:15:15.391 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2022-03-20T15:15:15.391 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/keyring 2022-03-20T15:15:15.449 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2022-03-20T15:15:15.450 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/ceph/ceph.conf 2022-03-20T15:15:15.465 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2022-03-20T15:15:15.465 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/ceph/keyring 2022-03-20T15:15:15.478 INFO:tasks.rook:Setting up client nodes... 2022-03-20T15:15:15.479 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph auth get-or-create client.a mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-20T15:15:16.217 INFO:teuthology.orchestra.run.smithi006.stdout:[client.a] 2022-03-20T15:15:16.218 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQAERTdiksUVBhAA9MCHQ5QWxY89PUs9QQXTpQ== 2022-03-20T15:15:16.241 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2022-03-20T15:15:16.242 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/ceph.client.a.keyring 2022-03-20T15:15:16.242 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/ceph.client.a.keyring 2022-03-20T15:15:16.265 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph auth get-or-create client.b mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-20T15:15:17.052 INFO:teuthology.orchestra.run.smithi006.stdout:[client.b] 2022-03-20T15:15:17.053 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQAERTdi+pNiNhAAI++7wjiTi+K7PE1djJqMTQ== 2022-03-20T15:15:17.077 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2022-03-20T15:15:17.078 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/ceph.client.b.keyring 2022-03-20T15:15:17.078 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0644 /etc/ceph/ceph.client.b.keyring 2022-03-20T15:15:17.145 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph auth get-or-create client.c mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-20T15:15:18.161 INFO:teuthology.orchestra.run.smithi006.stdout:[client.c] 2022-03-20T15:15:18.161 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQAFRTdigIKiLRAAkauniVQymlWh2iPgxxOsUQ== 2022-03-20T15:15:18.186 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2022-03-20T15:15:18.187 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/ceph/ceph.client.c.keyring 2022-03-20T15:15:18.187 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0644 /etc/ceph/ceph.client.c.keyring 2022-03-20T15:15:18.206 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd dump --format=json 2022-03-20T15:15:18.749 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:15:18.749 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":41,"fsid":"99bab31c-0e4a-4080-8a23-67b926f7eb20","created":"2022-03-20T15:09:14.387013+0000","modified":"2022-03-20T15:15:15.463819+0000","last_up_change":"2022-03-20T15:15:10.972688+0000","last_in_change":"2022-03-20T15:13:42.028162+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-20T15:14:20.806251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"19ccea14-1d73-48e3-8968-2309e5cdd0fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.140:6800","nonce":3463423944},{"type":"v1","addr":"10.248.47.140:6801","nonce":3463423944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.140:6802","nonce":3463423944},{"type":"v1","addr":"10.248.47.140:6803","nonce":3463423944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.140:6806","nonce":3463423944},{"type":"v1","addr":"10.248.47.140:6807","nonce":3463423944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.140:6804","nonce":3463423944},{"type":"v1","addr":"10.248.47.140:6805","nonce":3463423944}]},"public_addr":"10.248.47.140:6801/3463423944","cluster_addr":"10.248.47.140:6803/3463423944","heartbeat_back_addr":"10.248.47.140:6807/3463423944","heartbeat_front_addr":"10.248.47.140:6805/3463423944","state":["exists","up"]},{"osd":1,"uuid":"5135c013-924c-4d48-aff8-c1512b5f32f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.140:6800","nonce":738281331},{"type":"v1","addr":"10.248.46.140:6801","nonce":738281331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.140:6802","nonce":738281331},{"type":"v1","addr":"10.248.46.140:6803","nonce":738281331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.140:6806","nonce":738281331},{"type":"v1","addr":"10.248.46.140:6807","nonce":738281331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.140:6804","nonce":738281331},{"type":"v1","addr":"10.248.46.140:6805","nonce":738281331}]},"public_addr":"10.248.46.140:6801/738281331","cluster_addr":"10.248.46.140:6803/738281331","heartbeat_back_addr":"10.248.46.140:6807/738281331","heartbeat_front_addr":"10.248.46.140:6805/738281331","state":["exists","up"]},{"osd":2,"uuid":"e9a9b3cc-a4cd-40ed-9d6d-4671a8a44839","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.207:6800","nonce":2665006682},{"type":"v1","addr":"10.248.42.207:6801","nonce":2665006682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.207:6802","nonce":2665006682},{"type":"v1","addr":"10.248.42.207:6803","nonce":2665006682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.207:6806","nonce":2665006682},{"type":"v1","addr":"10.248.42.207:6807","nonce":2665006682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.207:6804","nonce":2665006682},{"type":"v1","addr":"10.248.42.207:6805","nonce":2665006682}]},"public_addr":"10.248.42.207:6801/2665006682","cluster_addr":"10.248.42.207:6803/2665006682","heartbeat_back_addr":"10.248.42.207:6807/2665006682","heartbeat_front_addr":"10.248.42.207:6805/2665006682","state":["exists","up"]},{"osd":3,"uuid":"fe8a3d5c-f572-43c4-9e8c-36f4bb28d56a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.141:6800","nonce":3217059015},{"type":"v1","addr":"10.248.47.141:6801","nonce":3217059015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.141:6802","nonce":3217059015},{"type":"v1","addr":"10.248.47.141:6803","nonce":3217059015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.141:6806","nonce":3217059015},{"type":"v1","addr":"10.248.47.141:6807","nonce":3217059015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.141:6804","nonce":3217059015},{"type":"v1","addr":"10.248.47.141:6805","nonce":3217059015}]},"public_addr":"10.248.47.141:6801/3217059015","cluster_addr":"10.248.47.141:6803/3217059015","heartbeat_back_addr":"10.248.47.141:6807/3217059015","heartbeat_front_addr":"10.248.47.141:6805/3217059015","state":["exists","up"]},{"osd":4,"uuid":"ba8bf741-57f3-449e-b3c8-351e8a4b3881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.141:6800","nonce":400501941},{"type":"v1","addr":"10.248.46.141:6801","nonce":400501941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.141:6802","nonce":400501941},{"type":"v1","addr":"10.248.46.141:6803","nonce":400501941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.141:6806","nonce":400501941},{"type":"v1","addr":"10.248.46.141:6807","nonce":400501941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.141:6804","nonce":400501941},{"type":"v1","addr":"10.248.46.141:6805","nonce":400501941}]},"public_addr":"10.248.46.141:6801/400501941","cluster_addr":"10.248.46.141:6803/400501941","heartbeat_back_addr":"10.248.46.141:6807/400501941","heartbeat_front_addr":"10.248.46.141:6805/400501941","state":["exists","up"]},{"osd":5,"uuid":"1935b0a1-27ca-4836-b47a-3b4028757814","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.143:6800","nonce":2604711588},{"type":"v1","addr":"10.248.46.143:6801","nonce":2604711588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.143:6802","nonce":2604711588},{"type":"v1","addr":"10.248.46.143:6803","nonce":2604711588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.143:6806","nonce":2604711588},{"type":"v1","addr":"10.248.46.143:6807","nonce":2604711588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.143:6804","nonce":2604711588},{"type":"v1","addr":"10.248.46.143:6805","nonce":2604711588}]},"public_addr":"10.248.46.143:6801/2604711588","cluster_addr":"10.248.46.143:6803/2604711588","heartbeat_back_addr":"10.248.46.143:6807/2604711588","heartbeat_front_addr":"10.248.46.143:6805/2604711588","state":["exists","up"]},{"osd":6,"uuid":"39195c3b-6ea4-4c8c-82d3-e7b7b58a66f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.142:6800","nonce":1278673811},{"type":"v1","addr":"10.248.46.142:6801","nonce":1278673811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.142:6802","nonce":1278673811},{"type":"v1","addr":"10.248.46.142:6803","nonce":1278673811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.142:6806","nonce":1278673811},{"type":"v1","addr":"10.248.46.142:6807","nonce":1278673811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.142:6804","nonce":1278673811},{"type":"v1","addr":"10.248.46.142:6805","nonce":1278673811}]},"public_addr":"10.248.46.142:6801/1278673811","cluster_addr":"10.248.46.142:6803/1278673811","heartbeat_back_addr":"10.248.46.142:6807/1278673811","heartbeat_front_addr":"10.248.46.142:6805/1278673811","state":["exists","up"]},{"osd":7,"uuid":"c532cd76-53bb-4d32-8445-aae145ac6d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.142:6800","nonce":1017196653},{"type":"v1","addr":"10.248.47.142:6801","nonce":1017196653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.142:6802","nonce":1017196653},{"type":"v1","addr":"10.248.47.142:6803","nonce":1017196653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.142:6806","nonce":1017196653},{"type":"v1","addr":"10.248.47.142:6807","nonce":1017196653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.142:6804","nonce":1017196653},{"type":"v1","addr":"10.248.47.142:6805","nonce":1017196653}]},"public_addr":"10.248.47.142:6801/1017196653","cluster_addr":"10.248.47.142:6803/1017196653","heartbeat_back_addr":"10.248.47.142:6807/1017196653","heartbeat_front_addr":"10.248.47.142:6805/1017196653","state":["exists","up"]},{"osd":8,"uuid":"79a1263e-e100-4cbc-9d22-1b35ec8f693f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.208:6800","nonce":2848503019},{"type":"v1","addr":"10.248.42.208:6801","nonce":2848503019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.208:6802","nonce":2848503019},{"type":"v1","addr":"10.248.42.208:6803","nonce":2848503019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.208:6806","nonce":2848503019},{"type":"v1","addr":"10.248.42.208:6807","nonce":2848503019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.208:6804","nonce":2848503019},{"type":"v1","addr":"10.248.42.208:6805","nonce":2848503019}]},"public_addr":"10.248.42.208:6801/2848503019","cluster_addr":"10.248.42.208:6803/2848503019","heartbeat_back_addr":"10.248.42.208:6807/2848503019","heartbeat_front_addr":"10.248.42.208:6805/2848503019","state":["exists","up"]},{"osd":9,"uuid":"c8c86598-426b-4daa-b8e1-49e34b5840b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.209:6800","nonce":4042027731},{"type":"v1","addr":"10.248.42.209:6801","nonce":4042027731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.209:6802","nonce":4042027731},{"type":"v1","addr":"10.248.42.209:6803","nonce":4042027731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.209:6806","nonce":4042027731},{"type":"v1","addr":"10.248.42.209:6807","nonce":4042027731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.209:6804","nonce":4042027731},{"type":"v1","addr":"10.248.42.209:6805","nonce":4042027731}]},"public_addr":"10.248.42.209:6801/4042027731","cluster_addr":"10.248.42.209:6803/4042027731","heartbeat_back_addr":"10.248.42.209:6807/4042027731","heartbeat_front_addr":"10.248.42.209:6805/4042027731","state":["exists","up"]},{"osd":10,"uuid":"816bbd44-dfae-4b22-bd8b-f4766e174f44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.211:6800","nonce":1761037137},{"type":"v1","addr":"10.248.42.211:6801","nonce":1761037137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.211:6802","nonce":1761037137},{"type":"v1","addr":"10.248.42.211:6803","nonce":1761037137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.211:6806","nonce":1761037137},{"type":"v1","addr":"10.248.42.211:6807","nonce":1761037137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.211:6804","nonce":1761037137},{"type":"v1","addr":"10.248.42.211:6805","nonce":1761037137}]},"public_addr":"10.248.42.211:6801/1761037137","cluster_addr":"10.248.42.211:6803/1761037137","heartbeat_back_addr":"10.248.42.211:6807/1761037137","heartbeat_front_addr":"10.248.42.211:6805/1761037137","state":["exists","up"]},{"osd":11,"uuid":"d4acd265-d87b-492f-9628-7f66fccbee5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.144:6800","nonce":4091731206},{"type":"v1","addr":"10.248.47.144:6801","nonce":4091731206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.144:6802","nonce":4091731206},{"type":"v1","addr":"10.248.47.144:6803","nonce":4091731206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.144:6806","nonce":4091731206},{"type":"v1","addr":"10.248.47.144:6807","nonce":4091731206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.144:6804","nonce":4091731206},{"type":"v1","addr":"10.248.47.144:6805","nonce":4091731206}]},"public_addr":"10.248.47.144:6801/4091731206","cluster_addr":"10.248.47.144:6803/4091731206","heartbeat_back_addr":"10.248.47.144:6807/4091731206","heartbeat_front_addr":"10.248.47.144:6805/4091731206","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T15:14:10.562337+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T15:14:30.557851+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T15:14:10.488739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T15:14:13.252113+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T15:14:30.452876+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T15:14:30.557112+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T15:14:29.037340+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T15:14:15.495201+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T15:14:19.333586+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T15:14:20.678416+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T15:15:09.031504+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T15:15:07.642669+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.248.46.133:6801/1582305423":"2022-03-21T15:12:29.825252+0000","10.248.46.133:6800/1582305423":"2022-03-21T15:12:29.825252+0000","10.248.46.133:0/1631789322":"2022-03-21T15:12:29.825252+0000","10.248.46.133:0/3792601828":"2022-03-21T15:12:29.825252+0000","10.248.46.133:0/613594625":"2022-03-21T15:12:29.825252+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-20T15:15:18.807 INFO:tasks.rook.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-03-20T15:14:20.806251+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'on', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-03-20T15:15:18.807 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd pool get .mgr pg_num 2022-03-20T15:15:19.646 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2022-03-20T15:15:19.665 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-03-20T15:15:19.666 INFO:tasks.rook.ceph_manager.ceph:waiting for mgr available 2022-03-20T15:15:19.667 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph mgr dump --format=json 2022-03-20T15:15:20.303 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:15:20.322 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":16,"active_gid":24197,"active_name":"a","active_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.133:6800","nonce":3001684104},{"type":"v1","addr":"10.248.46.133:6801","nonce":3001684104}]},"active_addr":"10.248.46.133:6801/3001684104","active_change":"2022-03-20T15:12:36.944801+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","prometheus","restful","rook"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"prometheus":"http://10.248.46.133:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":3,"active_clients":[{"addrvec":[{"type":"v2","addr":"10.248.46.133:0","nonce":3932218571}]},{"addrvec":[{"type":"v2","addr":"10.248.46.133:0","nonce":4209871800}]},{"addrvec":[{"type":"v2","addr":"10.248.46.133:0","nonce":2409261092}]},{"addrvec":[{"type":"v2","addr":"10.248.46.133:0","nonce":2445226432}]}]}} 2022-03-20T15:15:20.327 INFO:tasks.rook.ceph_manager.ceph:mgr available! 2022-03-20T15:15:20.327 INFO:tasks.rook.ceph_manager.ceph:waiting for all up 2022-03-20T15:15:20.328 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd dump --format=json 2022-03-20T15:15:20.843 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:15:20.843 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":41,"fsid":"99bab31c-0e4a-4080-8a23-67b926f7eb20","created":"2022-03-20T15:09:14.387013+0000","modified":"2022-03-20T15:15:15.463819+0000","last_up_change":"2022-03-20T15:15:10.972688+0000","last_in_change":"2022-03-20T15:13:42.028162+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-20T15:14:20.806251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"19ccea14-1d73-48e3-8968-2309e5cdd0fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.140:6800","nonce":3463423944},{"type":"v1","addr":"10.248.47.140:6801","nonce":3463423944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.140:6802","nonce":3463423944},{"type":"v1","addr":"10.248.47.140:6803","nonce":3463423944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.140:6806","nonce":3463423944},{"type":"v1","addr":"10.248.47.140:6807","nonce":3463423944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.140:6804","nonce":3463423944},{"type":"v1","addr":"10.248.47.140:6805","nonce":3463423944}]},"public_addr":"10.248.47.140:6801/3463423944","cluster_addr":"10.248.47.140:6803/3463423944","heartbeat_back_addr":"10.248.47.140:6807/3463423944","heartbeat_front_addr":"10.248.47.140:6805/3463423944","state":["exists","up"]},{"osd":1,"uuid":"5135c013-924c-4d48-aff8-c1512b5f32f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.140:6800","nonce":738281331},{"type":"v1","addr":"10.248.46.140:6801","nonce":738281331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.140:6802","nonce":738281331},{"type":"v1","addr":"10.248.46.140:6803","nonce":738281331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.140:6806","nonce":738281331},{"type":"v1","addr":"10.248.46.140:6807","nonce":738281331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.140:6804","nonce":738281331},{"type":"v1","addr":"10.248.46.140:6805","nonce":738281331}]},"public_addr":"10.248.46.140:6801/738281331","cluster_addr":"10.248.46.140:6803/738281331","heartbeat_back_addr":"10.248.46.140:6807/738281331","heartbeat_front_addr":"10.248.46.140:6805/738281331","state":["exists","up"]},{"osd":2,"uuid":"e9a9b3cc-a4cd-40ed-9d6d-4671a8a44839","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.207:6800","nonce":2665006682},{"type":"v1","addr":"10.248.42.207:6801","nonce":2665006682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.207:6802","nonce":2665006682},{"type":"v1","addr":"10.248.42.207:6803","nonce":2665006682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.207:6806","nonce":2665006682},{"type":"v1","addr":"10.248.42.207:6807","nonce":2665006682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.207:6804","nonce":2665006682},{"type":"v1","addr":"10.248.42.207:6805","nonce":2665006682}]},"public_addr":"10.248.42.207:6801/2665006682","cluster_addr":"10.248.42.207:6803/2665006682","heartbeat_back_addr":"10.248.42.207:6807/2665006682","heartbeat_front_addr":"10.248.42.207:6805/2665006682","state":["exists","up"]},{"osd":3,"uuid":"fe8a3d5c-f572-43c4-9e8c-36f4bb28d56a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.141:6800","nonce":3217059015},{"type":"v1","addr":"10.248.47.141:6801","nonce":3217059015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.141:6802","nonce":3217059015},{"type":"v1","addr":"10.248.47.141:6803","nonce":3217059015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.141:6806","nonce":3217059015},{"type":"v1","addr":"10.248.47.141:6807","nonce":3217059015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.141:6804","nonce":3217059015},{"type":"v1","addr":"10.248.47.141:6805","nonce":3217059015}]},"public_addr":"10.248.47.141:6801/3217059015","cluster_addr":"10.248.47.141:6803/3217059015","heartbeat_back_addr":"10.248.47.141:6807/3217059015","heartbeat_front_addr":"10.248.47.141:6805/3217059015","state":["exists","up"]},{"osd":4,"uuid":"ba8bf741-57f3-449e-b3c8-351e8a4b3881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.141:6800","nonce":400501941},{"type":"v1","addr":"10.248.46.141:6801","nonce":400501941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.141:6802","nonce":400501941},{"type":"v1","addr":"10.248.46.141:6803","nonce":400501941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.141:6806","nonce":400501941},{"type":"v1","addr":"10.248.46.141:6807","nonce":400501941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.141:6804","nonce":400501941},{"type":"v1","addr":"10.248.46.141:6805","nonce":400501941}]},"public_addr":"10.248.46.141:6801/400501941","cluster_addr":"10.248.46.141:6803/400501941","heartbeat_back_addr":"10.248.46.141:6807/400501941","heartbeat_front_addr":"10.248.46.141:6805/400501941","state":["exists","up"]},{"osd":5,"uuid":"1935b0a1-27ca-4836-b47a-3b4028757814","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.143:6800","nonce":2604711588},{"type":"v1","addr":"10.248.46.143:6801","nonce":2604711588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.143:6802","nonce":2604711588},{"type":"v1","addr":"10.248.46.143:6803","nonce":2604711588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.143:6806","nonce":2604711588},{"type":"v1","addr":"10.248.46.143:6807","nonce":2604711588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.143:6804","nonce":2604711588},{"type":"v1","addr":"10.248.46.143:6805","nonce":2604711588}]},"public_addr":"10.248.46.143:6801/2604711588","cluster_addr":"10.248.46.143:6803/2604711588","heartbeat_back_addr":"10.248.46.143:6807/2604711588","heartbeat_front_addr":"10.248.46.143:6805/2604711588","state":["exists","up"]},{"osd":6,"uuid":"39195c3b-6ea4-4c8c-82d3-e7b7b58a66f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.142:6800","nonce":1278673811},{"type":"v1","addr":"10.248.46.142:6801","nonce":1278673811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.142:6802","nonce":1278673811},{"type":"v1","addr":"10.248.46.142:6803","nonce":1278673811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.142:6806","nonce":1278673811},{"type":"v1","addr":"10.248.46.142:6807","nonce":1278673811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.142:6804","nonce":1278673811},{"type":"v1","addr":"10.248.46.142:6805","nonce":1278673811}]},"public_addr":"10.248.46.142:6801/1278673811","cluster_addr":"10.248.46.142:6803/1278673811","heartbeat_back_addr":"10.248.46.142:6807/1278673811","heartbeat_front_addr":"10.248.46.142:6805/1278673811","state":["exists","up"]},{"osd":7,"uuid":"c532cd76-53bb-4d32-8445-aae145ac6d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.142:6800","nonce":1017196653},{"type":"v1","addr":"10.248.47.142:6801","nonce":1017196653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.142:6802","nonce":1017196653},{"type":"v1","addr":"10.248.47.142:6803","nonce":1017196653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.142:6806","nonce":1017196653},{"type":"v1","addr":"10.248.47.142:6807","nonce":1017196653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.142:6804","nonce":1017196653},{"type":"v1","addr":"10.248.47.142:6805","nonce":1017196653}]},"public_addr":"10.248.47.142:6801/1017196653","cluster_addr":"10.248.47.142:6803/1017196653","heartbeat_back_addr":"10.248.47.142:6807/1017196653","heartbeat_front_addr":"10.248.47.142:6805/1017196653","state":["exists","up"]},{"osd":8,"uuid":"79a1263e-e100-4cbc-9d22-1b35ec8f693f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.208:6800","nonce":2848503019},{"type":"v1","addr":"10.248.42.208:6801","nonce":2848503019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.208:6802","nonce":2848503019},{"type":"v1","addr":"10.248.42.208:6803","nonce":2848503019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.208:6806","nonce":2848503019},{"type":"v1","addr":"10.248.42.208:6807","nonce":2848503019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.208:6804","nonce":2848503019},{"type":"v1","addr":"10.248.42.208:6805","nonce":2848503019}]},"public_addr":"10.248.42.208:6801/2848503019","cluster_addr":"10.248.42.208:6803/2848503019","heartbeat_back_addr":"10.248.42.208:6807/2848503019","heartbeat_front_addr":"10.248.42.208:6805/2848503019","state":["exists","up"]},{"osd":9,"uuid":"c8c86598-426b-4daa-b8e1-49e34b5840b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.209:6800","nonce":4042027731},{"type":"v1","addr":"10.248.42.209:6801","nonce":4042027731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.209:6802","nonce":4042027731},{"type":"v1","addr":"10.248.42.209:6803","nonce":4042027731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.209:6806","nonce":4042027731},{"type":"v1","addr":"10.248.42.209:6807","nonce":4042027731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.209:6804","nonce":4042027731},{"type":"v1","addr":"10.248.42.209:6805","nonce":4042027731}]},"public_addr":"10.248.42.209:6801/4042027731","cluster_addr":"10.248.42.209:6803/4042027731","heartbeat_back_addr":"10.248.42.209:6807/4042027731","heartbeat_front_addr":"10.248.42.209:6805/4042027731","state":["exists","up"]},{"osd":10,"uuid":"816bbd44-dfae-4b22-bd8b-f4766e174f44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.211:6800","nonce":1761037137},{"type":"v1","addr":"10.248.42.211:6801","nonce":1761037137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.211:6802","nonce":1761037137},{"type":"v1","addr":"10.248.42.211:6803","nonce":1761037137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.211:6806","nonce":1761037137},{"type":"v1","addr":"10.248.42.211:6807","nonce":1761037137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.211:6804","nonce":1761037137},{"type":"v1","addr":"10.248.42.211:6805","nonce":1761037137}]},"public_addr":"10.248.42.211:6801/1761037137","cluster_addr":"10.248.42.211:6803/1761037137","heartbeat_back_addr":"10.248.42.211:6807/1761037137","heartbeat_front_addr":"10.248.42.211:6805/1761037137","state":["exists","up"]},{"osd":11,"uuid":"d4acd265-d87b-492f-9628-7f66fccbee5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.144:6800","nonce":4091731206},{"type":"v1","addr":"10.248.47.144:6801","nonce":4091731206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.144:6802","nonce":4091731206},{"type":"v1","addr":"10.248.47.144:6803","nonce":4091731206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.144:6806","nonce":4091731206},{"type":"v1","addr":"10.248.47.144:6807","nonce":4091731206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.144:6804","nonce":4091731206},{"type":"v1","addr":"10.248.47.144:6805","nonce":4091731206}]},"public_addr":"10.248.47.144:6801/4091731206","cluster_addr":"10.248.47.144:6803/4091731206","heartbeat_back_addr":"10.248.47.144:6807/4091731206","heartbeat_front_addr":"10.248.47.144:6805/4091731206","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T15:14:10.562337+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T15:14:30.557851+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T15:14:10.488739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T15:14:13.252113+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T15:14:30.452876+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T15:14:30.557112+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T15:14:29.037340+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T15:14:15.495201+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T15:14:19.333586+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T15:14:20.678416+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T15:15:09.031504+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T15:15:07.642669+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.248.46.133:6801/1582305423":"2022-03-21T15:12:29.825252+0000","10.248.46.133:6800/1582305423":"2022-03-21T15:12:29.825252+0000","10.248.46.133:0/1631789322":"2022-03-21T15:12:29.825252+0000","10.248.46.133:0/3792601828":"2022-03-21T15:12:29.825252+0000","10.248.46.133:0/613594625":"2022-03-21T15:12:29.825252+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-20T15:15:20.864 INFO:tasks.rook.ceph_manager.ceph:all up! 2022-03-20T15:15:20.864 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd dump --format=json 2022-03-20T15:15:21.389 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:15:21.423 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":41,"fsid":"99bab31c-0e4a-4080-8a23-67b926f7eb20","created":"2022-03-20T15:09:14.387013+0000","modified":"2022-03-20T15:15:15.463819+0000","last_up_change":"2022-03-20T15:15:10.972688+0000","last_in_change":"2022-03-20T15:13:42.028162+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-20T15:14:20.806251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"19ccea14-1d73-48e3-8968-2309e5cdd0fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.140:6800","nonce":3463423944},{"type":"v1","addr":"10.248.47.140:6801","nonce":3463423944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.140:6802","nonce":3463423944},{"type":"v1","addr":"10.248.47.140:6803","nonce":3463423944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.140:6806","nonce":3463423944},{"type":"v1","addr":"10.248.47.140:6807","nonce":3463423944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.140:6804","nonce":3463423944},{"type":"v1","addr":"10.248.47.140:6805","nonce":3463423944}]},"public_addr":"10.248.47.140:6801/3463423944","cluster_addr":"10.248.47.140:6803/3463423944","heartbeat_back_addr":"10.248.47.140:6807/3463423944","heartbeat_front_addr":"10.248.47.140:6805/3463423944","state":["exists","up"]},{"osd":1,"uuid":"5135c013-924c-4d48-aff8-c1512b5f32f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.140:6800","nonce":738281331},{"type":"v1","addr":"10.248.46.140:6801","nonce":738281331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.140:6802","nonce":738281331},{"type":"v1","addr":"10.248.46.140:6803","nonce":738281331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.140:6806","nonce":738281331},{"type":"v1","addr":"10.248.46.140:6807","nonce":738281331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.140:6804","nonce":738281331},{"type":"v1","addr":"10.248.46.140:6805","nonce":738281331}]},"public_addr":"10.248.46.140:6801/738281331","cluster_addr":"10.248.46.140:6803/738281331","heartbeat_back_addr":"10.248.46.140:6807/738281331","heartbeat_front_addr":"10.248.46.140:6805/738281331","state":["exists","up"]},{"osd":2,"uuid":"e9a9b3cc-a4cd-40ed-9d6d-4671a8a44839","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.207:6800","nonce":2665006682},{"type":"v1","addr":"10.248.42.207:6801","nonce":2665006682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.207:6802","nonce":2665006682},{"type":"v1","addr":"10.248.42.207:6803","nonce":2665006682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.207:6806","nonce":2665006682},{"type":"v1","addr":"10.248.42.207:6807","nonce":2665006682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.207:6804","nonce":2665006682},{"type":"v1","addr":"10.248.42.207:6805","nonce":2665006682}]},"public_addr":"10.248.42.207:6801/2665006682","cluster_addr":"10.248.42.207:6803/2665006682","heartbeat_back_addr":"10.248.42.207:6807/2665006682","heartbeat_front_addr":"10.248.42.207:6805/2665006682","state":["exists","up"]},{"osd":3,"uuid":"fe8a3d5c-f572-43c4-9e8c-36f4bb28d56a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.141:6800","nonce":3217059015},{"type":"v1","addr":"10.248.47.141:6801","nonce":3217059015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.141:6802","nonce":3217059015},{"type":"v1","addr":"10.248.47.141:6803","nonce":3217059015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.141:6806","nonce":3217059015},{"type":"v1","addr":"10.248.47.141:6807","nonce":3217059015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.141:6804","nonce":3217059015},{"type":"v1","addr":"10.248.47.141:6805","nonce":3217059015}]},"public_addr":"10.248.47.141:6801/3217059015","cluster_addr":"10.248.47.141:6803/3217059015","heartbeat_back_addr":"10.248.47.141:6807/3217059015","heartbeat_front_addr":"10.248.47.141:6805/3217059015","state":["exists","up"]},{"osd":4,"uuid":"ba8bf741-57f3-449e-b3c8-351e8a4b3881","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.141:6800","nonce":400501941},{"type":"v1","addr":"10.248.46.141:6801","nonce":400501941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.141:6802","nonce":400501941},{"type":"v1","addr":"10.248.46.141:6803","nonce":400501941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.141:6806","nonce":400501941},{"type":"v1","addr":"10.248.46.141:6807","nonce":400501941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.141:6804","nonce":400501941},{"type":"v1","addr":"10.248.46.141:6805","nonce":400501941}]},"public_addr":"10.248.46.141:6801/400501941","cluster_addr":"10.248.46.141:6803/400501941","heartbeat_back_addr":"10.248.46.141:6807/400501941","heartbeat_front_addr":"10.248.46.141:6805/400501941","state":["exists","up"]},{"osd":5,"uuid":"1935b0a1-27ca-4836-b47a-3b4028757814","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.143:6800","nonce":2604711588},{"type":"v1","addr":"10.248.46.143:6801","nonce":2604711588}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.143:6802","nonce":2604711588},{"type":"v1","addr":"10.248.46.143:6803","nonce":2604711588}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.143:6806","nonce":2604711588},{"type":"v1","addr":"10.248.46.143:6807","nonce":2604711588}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.143:6804","nonce":2604711588},{"type":"v1","addr":"10.248.46.143:6805","nonce":2604711588}]},"public_addr":"10.248.46.143:6801/2604711588","cluster_addr":"10.248.46.143:6803/2604711588","heartbeat_back_addr":"10.248.46.143:6807/2604711588","heartbeat_front_addr":"10.248.46.143:6805/2604711588","state":["exists","up"]},{"osd":6,"uuid":"39195c3b-6ea4-4c8c-82d3-e7b7b58a66f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.142:6800","nonce":1278673811},{"type":"v1","addr":"10.248.46.142:6801","nonce":1278673811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.142:6802","nonce":1278673811},{"type":"v1","addr":"10.248.46.142:6803","nonce":1278673811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.142:6806","nonce":1278673811},{"type":"v1","addr":"10.248.46.142:6807","nonce":1278673811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.46.142:6804","nonce":1278673811},{"type":"v1","addr":"10.248.46.142:6805","nonce":1278673811}]},"public_addr":"10.248.46.142:6801/1278673811","cluster_addr":"10.248.46.142:6803/1278673811","heartbeat_back_addr":"10.248.46.142:6807/1278673811","heartbeat_front_addr":"10.248.46.142:6805/1278673811","state":["exists","up"]},{"osd":7,"uuid":"c532cd76-53bb-4d32-8445-aae145ac6d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.142:6800","nonce":1017196653},{"type":"v1","addr":"10.248.47.142:6801","nonce":1017196653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.142:6802","nonce":1017196653},{"type":"v1","addr":"10.248.47.142:6803","nonce":1017196653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.142:6806","nonce":1017196653},{"type":"v1","addr":"10.248.47.142:6807","nonce":1017196653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.142:6804","nonce":1017196653},{"type":"v1","addr":"10.248.47.142:6805","nonce":1017196653}]},"public_addr":"10.248.47.142:6801/1017196653","cluster_addr":"10.248.47.142:6803/1017196653","heartbeat_back_addr":"10.248.47.142:6807/1017196653","heartbeat_front_addr":"10.248.47.142:6805/1017196653","state":["exists","up"]},{"osd":8,"uuid":"79a1263e-e100-4cbc-9d22-1b35ec8f693f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.208:6800","nonce":2848503019},{"type":"v1","addr":"10.248.42.208:6801","nonce":2848503019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.208:6802","nonce":2848503019},{"type":"v1","addr":"10.248.42.208:6803","nonce":2848503019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.208:6806","nonce":2848503019},{"type":"v1","addr":"10.248.42.208:6807","nonce":2848503019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.208:6804","nonce":2848503019},{"type":"v1","addr":"10.248.42.208:6805","nonce":2848503019}]},"public_addr":"10.248.42.208:6801/2848503019","cluster_addr":"10.248.42.208:6803/2848503019","heartbeat_back_addr":"10.248.42.208:6807/2848503019","heartbeat_front_addr":"10.248.42.208:6805/2848503019","state":["exists","up"]},{"osd":9,"uuid":"c8c86598-426b-4daa-b8e1-49e34b5840b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.209:6800","nonce":4042027731},{"type":"v1","addr":"10.248.42.209:6801","nonce":4042027731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.209:6802","nonce":4042027731},{"type":"v1","addr":"10.248.42.209:6803","nonce":4042027731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.209:6806","nonce":4042027731},{"type":"v1","addr":"10.248.42.209:6807","nonce":4042027731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.209:6804","nonce":4042027731},{"type":"v1","addr":"10.248.42.209:6805","nonce":4042027731}]},"public_addr":"10.248.42.209:6801/4042027731","cluster_addr":"10.248.42.209:6803/4042027731","heartbeat_back_addr":"10.248.42.209:6807/4042027731","heartbeat_front_addr":"10.248.42.209:6805/4042027731","state":["exists","up"]},{"osd":10,"uuid":"816bbd44-dfae-4b22-bd8b-f4766e174f44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.211:6800","nonce":1761037137},{"type":"v1","addr":"10.248.42.211:6801","nonce":1761037137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.211:6802","nonce":1761037137},{"type":"v1","addr":"10.248.42.211:6803","nonce":1761037137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.211:6806","nonce":1761037137},{"type":"v1","addr":"10.248.42.211:6807","nonce":1761037137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.42.211:6804","nonce":1761037137},{"type":"v1","addr":"10.248.42.211:6805","nonce":1761037137}]},"public_addr":"10.248.42.211:6801/1761037137","cluster_addr":"10.248.42.211:6803/1761037137","heartbeat_back_addr":"10.248.42.211:6807/1761037137","heartbeat_front_addr":"10.248.42.211:6805/1761037137","state":["exists","up"]},{"osd":11,"uuid":"d4acd265-d87b-492f-9628-7f66fccbee5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.144:6800","nonce":4091731206},{"type":"v1","addr":"10.248.47.144:6801","nonce":4091731206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.144:6802","nonce":4091731206},{"type":"v1","addr":"10.248.47.144:6803","nonce":4091731206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.144:6806","nonce":4091731206},{"type":"v1","addr":"10.248.47.144:6807","nonce":4091731206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.47.144:6804","nonce":4091731206},{"type":"v1","addr":"10.248.47.144:6805","nonce":4091731206}]},"public_addr":"10.248.47.144:6801/4091731206","cluster_addr":"10.248.47.144:6803/4091731206","heartbeat_back_addr":"10.248.47.144:6807/4091731206","heartbeat_front_addr":"10.248.47.144:6805/4091731206","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T15:14:10.562337+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T15:14:30.557851+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T15:14:10.488739+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T15:14:13.252113+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T15:14:30.452876+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T15:14:30.557112+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T15:14:29.037340+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T15:14:15.495201+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T15:14:19.333586+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T15:14:20.678416+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T15:15:09.031504+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T15:15:07.642669+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.248.46.133:6801/1582305423":"2022-03-21T15:12:29.825252+0000","10.248.46.133:6800/1582305423":"2022-03-21T15:12:29.825252+0000","10.248.46.133:0/1631789322":"2022-03-21T15:12:29.825252+0000","10.248.46.133:0/3792601828":"2022-03-21T15:12:29.825252+0000","10.248.46.133:0/613594625":"2022-03-21T15:12:29.825252+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-20T15:15:21.425 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph tell osd.0 flush_pg_stats 2022-03-20T15:15:21.425 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph tell osd.1 flush_pg_stats 2022-03-20T15:15:21.426 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph tell osd.2 flush_pg_stats 2022-03-20T15:15:21.426 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph tell osd.3 flush_pg_stats 2022-03-20T15:15:21.427 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph tell osd.4 flush_pg_stats 2022-03-20T15:15:21.428 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph tell osd.5 flush_pg_stats 2022-03-20T15:15:21.428 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph tell osd.6 flush_pg_stats 2022-03-20T15:15:21.429 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph tell osd.7 flush_pg_stats 2022-03-20T15:15:21.430 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph tell osd.8 flush_pg_stats 2022-03-20T15:15:21.430 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph tell osd.9 flush_pg_stats 2022-03-20T15:15:21.431 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph tell osd.10 flush_pg_stats 2022-03-20T15:15:21.432 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph tell osd.11 flush_pg_stats 2022-03-20T15:15:21.944 INFO:teuthology.orchestra.run.smithi006.stdout:77309411343 2022-03-20T15:15:21.945 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd last-stat-seq osd.0 2022-03-20T15:15:22.286 INFO:teuthology.orchestra.run.smithi006.stdout:124554051596 2022-03-20T15:15:22.287 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd last-stat-seq osd.4 2022-03-20T15:15:22.290 INFO:teuthology.orchestra.run.smithi006.stdout:81604378643 2022-03-20T15:15:22.290 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd last-stat-seq osd.2 2022-03-20T15:15:22.297 INFO:teuthology.orchestra.run.smithi006.stdout:98784247823 2022-03-20T15:15:22.297 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd last-stat-seq osd.9 2022-03-20T15:15:22.306 INFO:teuthology.orchestra.run.smithi006.stdout:120259084301 2022-03-20T15:15:22.306 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd last-stat-seq osd.6 2022-03-20T15:15:22.313 INFO:teuthology.orchestra.run.smithi006.stdout:90194313231 2022-03-20T15:15:22.314 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd last-stat-seq osd.7 2022-03-20T15:15:22.317 INFO:teuthology.orchestra.run.smithi006.stdout:124554051595 2022-03-20T15:15:22.318 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd last-stat-seq osd.5 2022-03-20T15:15:22.320 INFO:teuthology.orchestra.run.smithi006.stdout:154618822662 2022-03-20T15:15:22.321 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd last-stat-seq osd.11 2022-03-20T15:15:22.323 INFO:teuthology.orchestra.run.smithi006.stdout:124554051597 2022-03-20T15:15:22.323 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd last-stat-seq osd.1 2022-03-20T15:15:22.339 INFO:teuthology.orchestra.run.smithi006.stdout:158913789957 2022-03-20T15:15:22.340 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd last-stat-seq osd.10 2022-03-20T15:15:22.481 INFO:teuthology.orchestra.run.smithi006.stdout:85899345936 2022-03-20T15:15:22.481 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd last-stat-seq osd.3 2022-03-20T15:15:22.525 INFO:teuthology.orchestra.run.smithi006.stdout:94489280527 2022-03-20T15:15:22.526 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd last-stat-seq osd.8 2022-03-20T15:15:22.728 INFO:teuthology.orchestra.run.smithi006.stdout:77309411342 2022-03-20T15:15:22.828 INFO:tasks.rook.ceph_manager.ceph:need seq 77309411343 got 77309411342 for osd.0 2022-03-20T15:15:22.964 INFO:teuthology.orchestra.run.smithi006.stdout:124554051595 2022-03-20T15:15:23.042 INFO:teuthology.orchestra.run.smithi006.stdout:98784247822 2022-03-20T15:15:23.113 INFO:tasks.rook.ceph_manager.ceph:need seq 124554051596 got 124554051595 for osd.4 2022-03-20T15:15:23.139 INFO:teuthology.orchestra.run.smithi006.stdout:90194313230 2022-03-20T15:15:23.198 INFO:teuthology.orchestra.run.smithi006.stdout:124554051594 2022-03-20T15:15:23.211 INFO:tasks.rook.ceph_manager.ceph:need seq 98784247823 got 98784247822 for osd.9 2022-03-20T15:15:23.220 INFO:tasks.rook.ceph_manager.ceph:need seq 90194313231 got 90194313230 for osd.7 2022-03-20T15:15:23.233 INFO:tasks.rook.ceph_manager.ceph:need seq 124554051595 got 124554051594 for osd.5 2022-03-20T15:15:23.288 INFO:teuthology.orchestra.run.smithi006.stdout:81604378642 2022-03-20T15:15:23.314 INFO:tasks.rook.ceph_manager.ceph:need seq 81604378643 got 81604378642 for osd.2 2022-03-20T15:15:23.345 INFO:teuthology.orchestra.run.smithi006.stdout:154618822661 2022-03-20T15:15:23.375 INFO:tasks.rook.ceph_manager.ceph:need seq 154618822662 got 154618822661 for osd.11 2022-03-20T15:15:23.411 INFO:teuthology.orchestra.run.smithi006.stdout:120259084300 2022-03-20T15:15:23.435 INFO:tasks.rook.ceph_manager.ceph:need seq 120259084301 got 120259084300 for osd.6 2022-03-20T15:15:23.465 INFO:teuthology.orchestra.run.smithi006.stdout:124554051596 2022-03-20T15:15:23.484 INFO:tasks.rook.ceph_manager.ceph:need seq 124554051597 got 124554051596 for osd.1 2022-03-20T15:15:23.521 INFO:teuthology.orchestra.run.smithi006.stdout:158913789955 2022-03-20T15:15:23.522 INFO:teuthology.orchestra.run.smithi006.stdout:85899345935 2022-03-20T15:15:23.536 INFO:tasks.rook.ceph_manager.ceph:need seq 158913789957 got 158913789955 for osd.10 2022-03-20T15:15:23.542 INFO:tasks.rook.ceph_manager.ceph:need seq 85899345936 got 85899345935 for osd.3 2022-03-20T15:15:23.574 INFO:teuthology.orchestra.run.smithi006.stdout:94489280525 2022-03-20T15:15:23.593 INFO:tasks.rook.ceph_manager.ceph:need seq 94489280527 got 94489280525 for osd.8 2022-03-20T15:15:23.829 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd last-stat-seq osd.0 2022-03-20T15:15:24.114 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd last-stat-seq osd.4 2022-03-20T15:15:24.212 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd last-stat-seq osd.9 2022-03-20T15:15:24.221 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd last-stat-seq osd.7 2022-03-20T15:15:24.235 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd last-stat-seq osd.5 2022-03-20T15:15:24.315 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd last-stat-seq osd.2 2022-03-20T15:15:24.373 INFO:teuthology.orchestra.run.smithi006.stdout:77309411344 2022-03-20T15:15:24.376 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd last-stat-seq osd.11 2022-03-20T15:15:24.437 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd last-stat-seq osd.6 2022-03-20T15:15:24.445 INFO:tasks.rook.ceph_manager.ceph:need seq 77309411343 got 77309411344 for osd.0 2022-03-20T15:15:24.445 DEBUG:teuthology.parallel:result is None 2022-03-20T15:15:24.485 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd last-stat-seq osd.1 2022-03-20T15:15:24.537 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd last-stat-seq osd.10 2022-03-20T15:15:24.543 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd last-stat-seq osd.3 2022-03-20T15:15:24.595 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph osd last-stat-seq osd.8 2022-03-20T15:15:24.682 INFO:teuthology.orchestra.run.smithi006.stdout:124554051597 2022-03-20T15:15:24.773 INFO:tasks.rook.ceph_manager.ceph:need seq 124554051596 got 124554051597 for osd.4 2022-03-20T15:15:24.773 DEBUG:teuthology.parallel:result is None 2022-03-20T15:15:24.892 INFO:teuthology.orchestra.run.smithi006.stdout:98784247823 2022-03-20T15:15:24.984 INFO:teuthology.orchestra.run.smithi006.stdout:124554051596 2022-03-20T15:15:25.024 INFO:teuthology.orchestra.run.smithi006.stdout:90194313231 2022-03-20T15:15:25.058 INFO:tasks.rook.ceph_manager.ceph:need seq 98784247823 got 98784247823 for osd.9 2022-03-20T15:15:25.058 DEBUG:teuthology.parallel:result is None 2022-03-20T15:15:25.062 INFO:tasks.rook.ceph_manager.ceph:need seq 124554051595 got 124554051596 for osd.5 2022-03-20T15:15:25.063 DEBUG:teuthology.parallel:result is None 2022-03-20T15:15:25.079 INFO:tasks.rook.ceph_manager.ceph:need seq 90194313231 got 90194313231 for osd.7 2022-03-20T15:15:25.079 DEBUG:teuthology.parallel:result is None 2022-03-20T15:15:25.202 INFO:teuthology.orchestra.run.smithi006.stdout:81604378643 2022-03-20T15:15:25.236 INFO:tasks.rook.ceph_manager.ceph:need seq 81604378643 got 81604378643 for osd.2 2022-03-20T15:15:25.236 DEBUG:teuthology.parallel:result is None 2022-03-20T15:15:25.301 INFO:teuthology.orchestra.run.smithi006.stdout:154618822662 2022-03-20T15:15:25.326 INFO:tasks.rook.ceph_manager.ceph:need seq 154618822662 got 154618822662 for osd.11 2022-03-20T15:15:25.326 DEBUG:teuthology.parallel:result is None 2022-03-20T15:15:25.347 INFO:teuthology.orchestra.run.smithi006.stdout:124554051598 2022-03-20T15:15:25.349 INFO:teuthology.orchestra.run.smithi006.stdout:120259084302 2022-03-20T15:15:25.368 INFO:tasks.rook.ceph_manager.ceph:need seq 124554051597 got 124554051598 for osd.1 2022-03-20T15:15:25.368 DEBUG:teuthology.parallel:result is None 2022-03-20T15:15:25.370 INFO:tasks.rook.ceph_manager.ceph:need seq 120259084301 got 120259084302 for osd.6 2022-03-20T15:15:25.371 DEBUG:teuthology.parallel:result is None 2022-03-20T15:15:25.418 INFO:teuthology.orchestra.run.smithi006.stdout:85899345936 2022-03-20T15:15:25.436 INFO:tasks.rook.ceph_manager.ceph:need seq 85899345936 got 85899345936 for osd.3 2022-03-20T15:15:25.437 DEBUG:teuthology.parallel:result is None 2022-03-20T15:15:25.483 INFO:teuthology.orchestra.run.smithi006.stdout:158913789957 2022-03-20T15:15:25.495 INFO:teuthology.orchestra.run.smithi006.stdout:94489280527 2022-03-20T15:15:25.503 INFO:tasks.rook.ceph_manager.ceph:need seq 158913789957 got 158913789957 for osd.10 2022-03-20T15:15:25.503 DEBUG:teuthology.parallel:result is None 2022-03-20T15:15:25.514 INFO:tasks.rook.ceph_manager.ceph:need seq 94489280527 got 94489280527 for osd.8 2022-03-20T15:15:25.515 DEBUG:teuthology.parallel:result is None 2022-03-20T15:15:25.515 INFO:tasks.rook.ceph_manager.ceph:waiting for clean 2022-03-20T15:15:25.515 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph pg dump --format=json 2022-03-20T15:15:26.020 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:15:26.022 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2022-03-20T15:15:26.044 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":115,"stamp":"2022-03-20T15:15:25.522253+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":996416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":68024,"kb_used_data":6552,"kb_used_omap":0,"kb_used_meta":61376,"kb_avail":1124873800,"statfs":{"total":1151940427776,"available":1151870771200,"internally_reserved":0,"allocated":6709248,"data_stored":3850968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":62849024},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003489"},"pg_stats":[{"pgid":"1.0","version":"31'117","reported_seq":234,"reported_epoch":41,"state":"active+clean","last_fresh":"2022-03-20T15:15:15.647104+0000","last_change":"2022-03-20T15:15:15.647104+0000","last_active":"2022-03-20T15:15:15.647104+0000","last_peered":"2022-03-20T15:15:15.647104+0000","last_clean":"2022-03-20T15:15:15.647104+0000","last_became_active":"2022-03-20T15:15:15.646407+0000","last_became_peered":"2022-03-20T15:15:15.646407+0000","last_unstale":"2022-03-20T15:15:15.647104+0000","last_undegraded":"2022-03-20T15:15:15.647104+0000","last_fullsized":"2022-03-20T15:15:15.647104+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-20T15:14:21.324848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-20T15:14:21.324848+0000","last_clean_scrub_stamp":"2022-03-20T15:14:21.324848+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T02:14:33.561325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":996416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,10],"acting":[8,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":996416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2600960,"data_stored":2583200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":10,"up_from":37,"seq":158913789957,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5872,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739280,"statfs":{"total":95995035648,"available":95989022720,"internally_reserved":0,"allocated":958464,"data_stored":717610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1990000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.171}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2470000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5820000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3440000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7490000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.468}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8939999999999999}]}]},{"osd":11,"up_from":36,"seq":154618822662,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5352,"kb_used_data":416,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739800,"statfs":{"total":95995035648,"available":95989555200,"internally_reserved":0,"allocated":425984,"data_stored":188682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4239999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5620000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.946}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2829999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4790000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4649999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5029999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3519999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.881}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.595}]}]},{"osd":5,"up_from":29,"seq":124554051596,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6000,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739152,"statfs":{"total":95995035648,"available":95988891648,"internally_reserved":0,"allocated":958464,"data_stored":717610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.298}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.613}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7390000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.431}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5189999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5820000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5549999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7010000000000001}]}]},{"osd":4,"up_from":29,"seq":124554051597,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5416,"kb_used_data":416,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739736,"statfs":{"total":95995035648,"available":95989489664,"internally_reserved":0,"allocated":425984,"data_stored":188682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.876}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5629999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2250000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6680000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.272}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.873}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.784}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4500000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051598,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5352,"kb_used_data":416,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739800,"statfs":{"total":95995035648,"available":95989555200,"internally_reserved":0,"allocated":425984,"data_stored":188682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2989999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5780000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.708}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6479999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9350000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.3100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2789999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.367}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5670000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.2029999999999998}]}]},{"osd":6,"up_from":28,"seq":120259084302,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5416,"kb_used_data":416,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739736,"statfs":{"total":95995035648,"available":95989489664,"internally_reserved":0,"allocated":425984,"data_stored":188682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1439999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6020000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.831}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2170000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9099999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.506}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.302}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.847}]}]},{"osd":2,"up_from":19,"seq":81604378644,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5992,"kb_used_data":416,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93739160,"statfs":{"total":95995035648,"available":95988899840,"internally_reserved":0,"allocated":425984,"data_stored":188682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 20 15:15:18 2022","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.9319999999999999,"5min":1.9319999999999999,"15min":1.9319999999999999},"last":2.4140000000000001},{"interface":"front","average":{"1min":1.016,"5min":1.016,"15min":1.016},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.544,"5min":1.544,"15min":1.544},"last":1.087}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":3,"last update":"Sun Mar 20 15:15:22 2022","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.083,"15min":1.083},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"min":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"max":{"1min":2.5190000000000001,"5min":2.5190000000000001,"15min":2.5190000000000001},"last":2.5190000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4830000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.157}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0409999999999999}]},{"osd":7,"last update":"Sun Mar 20 15:15:22 2022","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"min":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"max":{"1min":2.274,"5min":2.274,"15min":2.274},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":2.1720000000000002,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":0.88100000000000001}]},{"osd":8,"last update":"Sun Mar 20 15:15:22 2022","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.423,"15min":1.423},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":3.4700000000000002,"5min":3.4700000000000002,"15min":3.4700000000000002},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.365,"5min":1.365,"15min":1.365},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.7609999999999999,"5min":1.7609999999999999,"15min":1.7609999999999999},"last":0.91800000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.363}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]}]},{"osd":9,"up_from":23,"seq":98784247824,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5416,"kb_used_data":416,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739736,"statfs":{"total":95995035648,"available":95989489664,"internally_reserved":0,"allocated":425984,"data_stored":188682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0129999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.052}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.085}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.151}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97199999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0389999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1759999999999999}]}]},{"osd":0,"up_from":18,"seq":77309411344,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5864,"kb_used_data":416,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93739288,"statfs":{"total":95995035648,"available":95989030912,"internally_reserved":0,"allocated":425984,"data_stored":188682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0680000000000001}]},{"osd":2,"last update":"Sun Mar 20 15:15:18 2022","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.103,"15min":1.103},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.641,"5min":1.641,"15min":1.641},"last":1.1240000000000001},{"interface":"front","average":{"1min":1.212,"5min":1.212,"15min":1.212},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.847,"5min":1.847,"15min":1.847},"last":0.90800000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.427}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7749999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.708}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9710000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0169999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1100000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.995}]}]},{"osd":7,"up_from":21,"seq":90194313231,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5416,"kb_used_data":416,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739736,"statfs":{"total":95995035648,"available":95989489664,"internally_reserved":0,"allocated":425984,"data_stored":188682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.177}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2330000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.448}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9830000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.327}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7030000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.946}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.825}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3679999999999999}]}]},{"osd":3,"up_from":20,"seq":85899345937,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5480,"kb_used_data":416,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739672,"statfs":{"total":95995035648,"available":95989424128,"internally_reserved":0,"allocated":425984,"data_stored":188682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 20 15:15:22 2022","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.083,"15min":1.083},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.788,"5min":1.788,"15min":1.788},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.502,"5min":1.502,"15min":1.502},"last":1.6899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":2,"last update":"Sun Mar 20 15:15:22 2022","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":3.1539999999999999,"5min":3.1539999999999999,"15min":3.1539999999999999},"last":1.05},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"min":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"max":{"1min":3.1970000000000001,"5min":3.1970000000000001,"15min":3.1970000000000001},"last":1.75}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.641}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.232}]},{"osd":7,"last update":"Sun Mar 20 15:15:22 2022","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":3.137,"5min":3.137,"15min":3.137},"last":1.3700000000000001},{"interface":"front","average":{"1min":1.6359999999999999,"5min":1.6359999999999999,"15min":1.6359999999999999},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":2.8820000000000001,"5min":2.8820000000000001,"15min":2.8820000000000001},"last":1.343}]},{"osd":8,"last update":"Sun Mar 20 15:15:22 2022","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.645,"15min":1.645},"min":{"1min":1.01,"5min":1.01,"15min":1.01},"max":{"1min":2.415,"5min":2.415,"15min":2.415},"last":1.4099999999999999},{"interface":"front","average":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":1.7190000000000001},"min":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"max":{"1min":2.5419999999999998,"5min":2.5419999999999998,"15min":2.5419999999999998},"last":1.3220000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4850000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5469999999999999}]}]},{"osd":8,"up_from":22,"seq":94489280527,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6448,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738704,"statfs":{"total":95995035648,"available":95988432896,"internally_reserved":0,"allocated":958464,"data_stored":717610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0009999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.254}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94499999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.78}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.024}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3340000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.798}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.754}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.728}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.881}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":467488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-20T15:15:26.046 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph pg dump --format=json 2022-03-20T15:15:26.584 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:15:26.586 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2022-03-20T15:15:26.612 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":115,"stamp":"2022-03-20T15:15:25.522253+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":996416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":68024,"kb_used_data":6552,"kb_used_omap":0,"kb_used_meta":61376,"kb_avail":1124873800,"statfs":{"total":1151940427776,"available":1151870771200,"internally_reserved":0,"allocated":6709248,"data_stored":3850968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":62849024},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003489"},"pg_stats":[{"pgid":"1.0","version":"31'117","reported_seq":234,"reported_epoch":41,"state":"active+clean","last_fresh":"2022-03-20T15:15:15.647104+0000","last_change":"2022-03-20T15:15:15.647104+0000","last_active":"2022-03-20T15:15:15.647104+0000","last_peered":"2022-03-20T15:15:15.647104+0000","last_clean":"2022-03-20T15:15:15.647104+0000","last_became_active":"2022-03-20T15:15:15.646407+0000","last_became_peered":"2022-03-20T15:15:15.646407+0000","last_unstale":"2022-03-20T15:15:15.647104+0000","last_undegraded":"2022-03-20T15:15:15.647104+0000","last_fullsized":"2022-03-20T15:15:15.647104+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-20T15:14:21.324848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-20T15:14:21.324848+0000","last_clean_scrub_stamp":"2022-03-20T15:14:21.324848+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-22T02:14:33.561325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":996416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,10],"acting":[8,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":996416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2600960,"data_stored":2583200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":10,"up_from":37,"seq":158913789957,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5872,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739280,"statfs":{"total":95995035648,"available":95989022720,"internally_reserved":0,"allocated":958464,"data_stored":717610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1990000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.171}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2470000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5820000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3440000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7490000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.468}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8939999999999999}]}]},{"osd":11,"up_from":36,"seq":154618822662,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5352,"kb_used_data":416,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739800,"statfs":{"total":95995035648,"available":95989555200,"internally_reserved":0,"allocated":425984,"data_stored":188682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4239999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5620000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.946}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2829999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4790000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4649999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5029999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3519999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.881}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.595}]}]},{"osd":5,"up_from":29,"seq":124554051596,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6000,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739152,"statfs":{"total":95995035648,"available":95988891648,"internally_reserved":0,"allocated":958464,"data_stored":717610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.298}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.613}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7390000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.431}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5189999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5820000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5549999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7010000000000001}]}]},{"osd":4,"up_from":29,"seq":124554051597,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5416,"kb_used_data":416,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739736,"statfs":{"total":95995035648,"available":95989489664,"internally_reserved":0,"allocated":425984,"data_stored":188682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.876}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5629999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2250000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6680000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.272}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.873}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.784}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4500000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051598,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5352,"kb_used_data":416,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739800,"statfs":{"total":95995035648,"available":95989555200,"internally_reserved":0,"allocated":425984,"data_stored":188682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2989999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5780000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.708}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6479999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9350000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.3100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2789999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.367}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5670000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.2029999999999998}]}]},{"osd":6,"up_from":28,"seq":120259084302,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5416,"kb_used_data":416,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739736,"statfs":{"total":95995035648,"available":95989489664,"internally_reserved":0,"allocated":425984,"data_stored":188682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1439999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6020000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.831}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2170000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9099999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.506}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.302}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.847}]}]},{"osd":2,"up_from":19,"seq":81604378644,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5992,"kb_used_data":416,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93739160,"statfs":{"total":95995035648,"available":95988899840,"internally_reserved":0,"allocated":425984,"data_stored":188682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 20 15:15:18 2022","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.9319999999999999,"5min":1.9319999999999999,"15min":1.9319999999999999},"last":2.4140000000000001},{"interface":"front","average":{"1min":1.016,"5min":1.016,"15min":1.016},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.544,"5min":1.544,"15min":1.544},"last":1.087}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":3,"last update":"Sun Mar 20 15:15:22 2022","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.083,"15min":1.083},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"min":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"max":{"1min":2.5190000000000001,"5min":2.5190000000000001,"15min":2.5190000000000001},"last":2.5190000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4830000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.157}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0409999999999999}]},{"osd":7,"last update":"Sun Mar 20 15:15:22 2022","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"min":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"max":{"1min":2.274,"5min":2.274,"15min":2.274},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":2.1720000000000002,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":0.88100000000000001}]},{"osd":8,"last update":"Sun Mar 20 15:15:22 2022","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.423,"15min":1.423},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":3.4700000000000002,"5min":3.4700000000000002,"15min":3.4700000000000002},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.365,"5min":1.365,"15min":1.365},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.7609999999999999,"5min":1.7609999999999999,"15min":1.7609999999999999},"last":0.91800000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.363}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]}]},{"osd":9,"up_from":23,"seq":98784247824,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5416,"kb_used_data":416,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739736,"statfs":{"total":95995035648,"available":95989489664,"internally_reserved":0,"allocated":425984,"data_stored":188682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0129999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.052}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.085}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.151}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97199999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0389999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1759999999999999}]}]},{"osd":0,"up_from":18,"seq":77309411344,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5864,"kb_used_data":416,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93739288,"statfs":{"total":95995035648,"available":95989030912,"internally_reserved":0,"allocated":425984,"data_stored":188682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0680000000000001}]},{"osd":2,"last update":"Sun Mar 20 15:15:18 2022","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.103,"15min":1.103},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.641,"5min":1.641,"15min":1.641},"last":1.1240000000000001},{"interface":"front","average":{"1min":1.212,"5min":1.212,"15min":1.212},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.847,"5min":1.847,"15min":1.847},"last":0.90800000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.427}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7749999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.708}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9710000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0169999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1100000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.995}]}]},{"osd":7,"up_from":21,"seq":90194313231,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5416,"kb_used_data":416,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739736,"statfs":{"total":95995035648,"available":95989489664,"internally_reserved":0,"allocated":425984,"data_stored":188682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.177}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2330000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.448}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9830000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.327}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7030000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.946}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.825}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3679999999999999}]}]},{"osd":3,"up_from":20,"seq":85899345937,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5480,"kb_used_data":416,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739672,"statfs":{"total":95995035648,"available":95989424128,"internally_reserved":0,"allocated":425984,"data_stored":188682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 20 15:15:22 2022","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.083,"15min":1.083},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.788,"5min":1.788,"15min":1.788},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.502,"5min":1.502,"15min":1.502},"last":1.6899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":2,"last update":"Sun Mar 20 15:15:22 2022","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":3.1539999999999999,"5min":3.1539999999999999,"15min":3.1539999999999999},"last":1.05},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"min":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"max":{"1min":3.1970000000000001,"5min":3.1970000000000001,"15min":3.1970000000000001},"last":1.75}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.641}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.232}]},{"osd":7,"last update":"Sun Mar 20 15:15:22 2022","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":3.137,"5min":3.137,"15min":3.137},"last":1.3700000000000001},{"interface":"front","average":{"1min":1.6359999999999999,"5min":1.6359999999999999,"15min":1.6359999999999999},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":2.8820000000000001,"5min":2.8820000000000001,"15min":2.8820000000000001},"last":1.343}]},{"osd":8,"last update":"Sun Mar 20 15:15:22 2022","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.645,"15min":1.645},"min":{"1min":1.01,"5min":1.01,"15min":1.01},"max":{"1min":2.415,"5min":2.415,"15min":2.415},"last":1.4099999999999999},{"interface":"front","average":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":1.7190000000000001},"min":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"max":{"1min":2.5419999999999998,"5min":2.5419999999999998,"15min":2.5419999999999998},"last":1.3220000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4850000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5469999999999999}]}]},{"osd":8,"up_from":22,"seq":94489280527,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6448,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738704,"statfs":{"total":95995035648,"available":95988432896,"internally_reserved":0,"allocated":958464,"data_stored":717610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0009999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.254}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94499999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.78}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.024}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3340000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.798}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.754}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.728}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.881}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":467488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-20T15:15:26.614 INFO:tasks.rook.ceph_manager.ceph:clean! 2022-03-20T15:15:26.614 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-03-20T15:15:26.614 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy 2022-03-20T15:15:26.615 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph health --format=json 2022-03-20T15:15:27.270 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:15:27.270 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-03-20T15:15:27.289 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy done 2022-03-20T15:15:27.290 INFO:tasks.rook:Rook complete, yielding 2022-03-20T15:15:27.290 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-20T15:15:27.301 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph -s 2022-03-20T15:15:27.921 INFO:teuthology.orchestra.run.smithi006.stdout: cluster: 2022-03-20T15:15:27.922 INFO:teuthology.orchestra.run.smithi006.stdout: id: 99bab31c-0e4a-4080-8a23-67b926f7eb20 2022-03-20T15:15:27.922 INFO:teuthology.orchestra.run.smithi006.stdout: health: HEALTH_OK 2022-03-20T15:15:27.922 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:15:27.923 INFO:teuthology.orchestra.run.smithi006.stdout: services: 2022-03-20T15:15:27.923 INFO:teuthology.orchestra.run.smithi006.stdout: mon: 3 daemons, quorum a,b,c (age 3m) 2022-03-20T15:15:27.923 INFO:teuthology.orchestra.run.smithi006.stdout: mgr: a(active, since 2m) 2022-03-20T15:15:27.923 INFO:teuthology.orchestra.run.smithi006.stdout: osd: 12 osds: 12 up (since 16s), 12 in (since 105s) 2022-03-20T15:15:27.923 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:15:27.924 INFO:teuthology.orchestra.run.smithi006.stdout: data: 2022-03-20T15:15:27.924 INFO:teuthology.orchestra.run.smithi006.stdout: pools: 1 pools, 1 pgs 2022-03-20T15:15:27.924 INFO:teuthology.orchestra.run.smithi006.stdout: objects: 2 objects, 577 KiB 2022-03-20T15:15:27.924 INFO:teuthology.orchestra.run.smithi006.stdout: usage: 66 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-20T15:15:27.924 INFO:teuthology.orchestra.run.smithi006.stdout: pgs: 1 active+clean 2022-03-20T15:15:27.925 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:15:27.925 INFO:teuthology.orchestra.run.smithi006.stdout: io: 2022-03-20T15:15:27.925 INFO:teuthology.orchestra.run.smithi006.stdout: recovery: 44 KiB/s, 0 objects/s 2022-03-20T15:15:27.925 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:15:27.944 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch status 2022-03-20T15:15:28.661 INFO:teuthology.orchestra.run.smithi006.stdout:Backend: rook 2022-03-20T15:15:28.662 INFO:teuthology.orchestra.run.smithi006.stdout:Available: Yes 2022-03-20T15:15:28.685 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ps 2022-03-20T15:15:29.297 INFO:teuthology.orchestra.run.smithi006.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-20T15:15:29.298 INFO:teuthology.orchestra.run.smithi006.stdout:crashcollector.crash smithi006 running (113s) 0s ago 113s - - 7c0c85ada2e4 2022-03-20T15:15:29.298 INFO:teuthology.orchestra.run.smithi006.stdout:crashcollector.crash smithi045 running (3m) 0s ago 3m - - 7c0c85ada2e4 2022-03-20T15:15:29.298 INFO:teuthology.orchestra.run.smithi006.stdout:crashcollector.crash smithi123 running (3m) 0s ago 3m - - 7c0c85ada2e4 2022-03-20T15:15:29.298 INFO:teuthology.orchestra.run.smithi006.stdout:mgr.a smithi006 running (3m) 0s ago 3m - - 7c0c85ada2e4 2022-03-20T15:15:29.298 INFO:teuthology.orchestra.run.smithi006.stdout:mon.a smithi045 running (6m) 0s ago 6m - - 7c0c85ada2e4 2022-03-20T15:15:29.299 INFO:teuthology.orchestra.run.smithi006.stdout:mon.b smithi123 running (5m) 0s ago 5m - - 7c0c85ada2e4 2022-03-20T15:15:29.299 INFO:teuthology.orchestra.run.smithi006.stdout:mon.c smithi006 running (4m) 0s ago 4m - - 7c0c85ada2e4 2022-03-20T15:15:29.299 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0 smithi045 running (116s) 0s ago 116s - - 7c0c85ada2e4 2022-03-20T15:15:29.299 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1 smithi006 running (113s) 0s ago 114s - - 7c0c85ada2e4 2022-03-20T15:15:29.300 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10 smithi123 running (40s) 0s ago 40s - - 7c0c85ada2e4 2022-03-20T15:15:29.300 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11 smithi045 running (42s) 0s ago 42s - - 7c0c85ada2e4 2022-03-20T15:15:29.300 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2 smithi123 running (111s) 0s ago 112s - - 7c0c85ada2e4 2022-03-20T15:15:29.300 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3 smithi045 running (112s) 0s ago 113s - - 7c0c85ada2e4 2022-03-20T15:15:29.300 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4 smithi006 running (108s) 0s ago 108s - - 7c0c85ada2e4 2022-03-20T15:15:29.301 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5 smithi006 running (105s) 0s ago 106s - - 7c0c85ada2e4 2022-03-20T15:15:29.301 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6 smithi006 running (107s) 0s ago 107s - - 7c0c85ada2e4 2022-03-20T15:15:29.301 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7 smithi045 running (104s) 0s ago 104s - - 7c0c85ada2e4 2022-03-20T15:15:29.301 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8 smithi123 running (102s) 0s ago 102s - - 7c0c85ada2e4 2022-03-20T15:15:29.302 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9 smithi123 running (101s) 0s ago 101s - - 7c0c85ada2e4 2022-03-20T15:15:29.320 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls 2022-03-20T15:15:30.007 INFO:teuthology.orchestra.run.smithi006.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-20T15:15:30.007 INFO:teuthology.orchestra.run.smithi006.stdout:crash 3/3 0s ago 3m * 2022-03-20T15:15:30.008 INFO:teuthology.orchestra.run.smithi006.stdout:mgr 1/1 0s ago 3m count:1 2022-03-20T15:15:30.008 INFO:teuthology.orchestra.run.smithi006.stdout:mon 3/3 0s ago 6m count:3 2022-03-20T15:15:30.008 INFO:teuthology.orchestra.run.smithi006.stdout:osd 24 0s ago 117s 2022-03-20T15:15:30.008 INFO:teuthology.orchestra.run.smithi006.stdout:osd.all-available-devices 0 0s ago - * 2022-03-20T15:15:30.029 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch device ls 2022-03-20T15:15:30.572 INFO:teuthology.orchestra.run.smithi006.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-20T15:15:30.573 INFO:teuthology.orchestra.run.smithi006.stdout:smithi006 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-20T15:15:30.573 INFO:teuthology.orchestra.run.smithi006.stdout:smithi006 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-20T15:15:30.573 INFO:teuthology.orchestra.run.smithi006.stdout:smithi006 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-20T15:15:30.574 INFO:teuthology.orchestra.run.smithi006.stdout:smithi006 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-20T15:15:30.574 INFO:teuthology.orchestra.run.smithi006.stdout:smithi045 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-20T15:15:30.574 INFO:teuthology.orchestra.run.smithi006.stdout:smithi045 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-20T15:15:30.575 INFO:teuthology.orchestra.run.smithi006.stdout:smithi045 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-20T15:15:30.575 INFO:teuthology.orchestra.run.smithi006.stdout:smithi045 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-20T15:15:30.576 INFO:teuthology.orchestra.run.smithi006.stdout:smithi123 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-20T15:15:30.576 INFO:teuthology.orchestra.run.smithi006.stdout:smithi123 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-20T15:15:30.577 INFO:teuthology.orchestra.run.smithi006.stdout:smithi123 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-20T15:15:30.577 INFO:teuthology.orchestra.run.smithi006.stdout:smithi123 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-20T15:15:30.590 INFO:teuthology.run_tasks:Running task exec... 2022-03-20T15:15:30.602 INFO:teuthology.task.exec:Executing custom commands... 2022-03-20T15:15:30.603 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi006.front.sepia.ceph.com 2022-03-20T15:15:30.603 DEBUG:teuthology.orchestra.run.smithi006:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'set -ex 2022-03-20T15:15:30.604 DEBUG:teuthology.orchestra.run.smithi006:> toolbox() { 2022-03-20T15:15:30.604 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- "$@" 2022-03-20T15:15:30.605 DEBUG:teuthology.orchestra.run.smithi006:> } 2022-03-20T15:15:30.606 DEBUG:teuthology.orchestra.run.smithi006:> orig_num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-20T15:15:30.606 DEBUG:teuthology.orchestra.run.smithi006:> toolbox ceph orch osd rm 0 --force 2022-03-20T15:15:30.607 DEBUG:teuthology.orchestra.run.smithi006:> removed_pv="" 2022-03-20T15:15:30.607 DEBUG:teuthology.orchestra.run.smithi006:> while [ "$removed_pv" = "" ] 2022-03-20T15:15:30.608 DEBUG:teuthology.orchestra.run.smithi006:> do 2022-03-20T15:15:30.608 DEBUG:teuthology.orchestra.run.smithi006:> removed_pv=`kubectl get pv | grep Released | cut -f1 -d " "` 2022-03-20T15:15:30.609 DEBUG:teuthology.orchestra.run.smithi006:> sleep 3s 2022-03-20T15:15:30.609 DEBUG:teuthology.orchestra.run.smithi006:> done 2022-03-20T15:15:30.609 DEBUG:teuthology.orchestra.run.smithi006:> target_path=`kubectl get pv $removed_pv -o jsonpath='"'"'{.spec.local.path}'"'"'` 2022-03-20T15:15:30.610 DEBUG:teuthology.orchestra.run.smithi006:> host=`echo $removed_pv | cut -f1 -d "-"` 2022-03-20T15:15:30.610 DEBUG:teuthology.orchestra.run.smithi006:> toolbox ceph orch device zap $host $target_path --force 2022-03-20T15:15:30.610 DEBUG:teuthology.orchestra.run.smithi006:> zap_completion="0" 2022-03-20T15:15:30.611 DEBUG:teuthology.orchestra.run.smithi006:> while [ "$zap_completion" = "0" ] 2022-03-20T15:15:30.611 DEBUG:teuthology.orchestra.run.smithi006:> do 2022-03-20T15:15:30.611 DEBUG:teuthology.orchestra.run.smithi006:> zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap -o jsonpath='"'"'{.status.succeeded.path}'"'"'` 2022-03-20T15:15:30.611 DEBUG:teuthology.orchestra.run.smithi006:> sleep 3s 2022-03-20T15:15:30.612 DEBUG:teuthology.orchestra.run.smithi006:> done 2022-03-20T15:15:30.612 DEBUG:teuthology.orchestra.run.smithi006:> kubectl patch pv $removed_pv -p '"'"'{"spec":{"claimRef": null}}'"'"' 2022-03-20T15:15:30.612 DEBUG:teuthology.orchestra.run.smithi006:> toolbox ceph orch apply osd --all-available-devices 2022-03-20T15:15:30.613 DEBUG:teuthology.orchestra.run.smithi006:> kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-20T15:15:30.613 DEBUG:teuthology.orchestra.run.smithi006:> num_osd="0" 2022-03-20T15:15:30.613 DEBUG:teuthology.orchestra.run.smithi006:> while [ "$num_osd" != "$orig_num_osd" ] 2022-03-20T15:15:30.614 DEBUG:teuthology.orchestra.run.smithi006:> do 2022-03-20T15:15:30.614 DEBUG:teuthology.orchestra.run.smithi006:> echo "waiting for osd to come back up" 2022-03-20T15:15:30.614 DEBUG:teuthology.orchestra.run.smithi006:> num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-20T15:15:30.615 DEBUG:teuthology.orchestra.run.smithi006:> sleep 30s 2022-03-20T15:15:30.615 DEBUG:teuthology.orchestra.run.smithi006:> done 2022-03-20T15:15:30.616 DEBUG:teuthology.orchestra.run.smithi006:> ' 2022-03-20T15:15:30.634 INFO:teuthology.orchestra.run.smithi006.stderr:++ toolbox ceph osd stat 2022-03-20T15:15:30.634 INFO:teuthology.orchestra.run.smithi006.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-20T15:15:30.635 INFO:teuthology.orchestra.run.smithi006.stderr:++ cut -f3 -d ' ' 2022-03-20T15:15:33.463 INFO:teuthology.orchestra.run.smithi006.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-20T15:15:34.095 INFO:teuthology.orchestra.run.smithi006.stderr:+ orig_num_osd=12 2022-03-20T15:15:34.095 INFO:teuthology.orchestra.run.smithi006.stderr:+ toolbox ceph orch osd rm 0 --force 2022-03-20T15:15:34.096 INFO:teuthology.orchestra.run.smithi006.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch osd rm 0 --force 2022-03-20T15:15:34.179 INFO:teuthology.orchestra.run.smithi006.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-20T15:15:34.801 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:34.795809 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-20T15:15:34.801 INFO:tasks.rook.operator.smithi006.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", AllowUnsupported: true}, 2022-03-20T15:15:34.802 INFO:tasks.rook.operator.smithi006.stdout: Storage: v1.StorageScopeSpec{ 2022-03-20T15:15:34.802 INFO:tasks.rook.operator.smithi006.stdout: ... // 3 identical fields 2022-03-20T15:15:34.802 INFO:tasks.rook.operator.smithi006.stdout: Config: nil, 2022-03-20T15:15:34.802 INFO:tasks.rook.operator.smithi006.stdout: Selection: {}, 2022-03-20T15:15:34.803 INFO:tasks.rook.operator.smithi006.stdout: StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-20T15:15:34.803 INFO:tasks.rook.operator.smithi006.stdout: ... // 2 identical elements 2022-03-20T15:15:34.803 INFO:tasks.rook.operator.smithi006.stdout: {Name: "smithi006-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi006-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-20T15:15:34.803 INFO:tasks.rook.operator.smithi006.stdout: {Name: "smithi006-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi006-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-03-20T15:15:34.804 INFO:tasks.rook.operator.smithi006.stdout:- { 2022-03-20T15:15:34.804 INFO:tasks.rook.operator.smithi006.stdout:- Name: "smithi045-lv-1", 2022-03-20T15:15:34.804 INFO:tasks.rook.operator.smithi006.stdout:- Count: 1, 2022-03-20T15:15:34.804 INFO:tasks.rook.operator.smithi006.stdout:- VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-20T15:15:34.804 INFO:tasks.rook.operator.smithi006.stdout:- }, 2022-03-20T15:15:34.805 INFO:tasks.rook.operator.smithi006.stdout: {Name: "smithi045-lv-2", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi045-lv-2", StorageClassName: &"scratch", ...}}}}, 2022-03-20T15:15:34.805 INFO:tasks.rook.operator.smithi006.stdout: {Name: "smithi045-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi045-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-20T15:15:34.805 INFO:tasks.rook.operator.smithi006.stdout: ... // 5 identical elements 2022-03-20T15:15:34.806 INFO:tasks.rook.operator.smithi006.stdout: }, 2022-03-20T15:15:34.806 INFO:tasks.rook.operator.smithi006.stdout: }, 2022-03-20T15:15:34.806 INFO:tasks.rook.operator.smithi006.stdout: Annotations: nil, 2022-03-20T15:15:34.806 INFO:tasks.rook.operator.smithi006.stdout: Labels: nil, 2022-03-20T15:15:34.807 INFO:tasks.rook.operator.smithi006.stdout: ... // 20 identical fields 2022-03-20T15:15:34.807 INFO:tasks.rook.operator.smithi006.stdout: } 2022-03-20T15:15:34.807 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:34.795968 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-20T15:15:34.807 INFO:tasks.rook.operator.smithi006.stdout:I0320 15:15:34.796351 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-20T15:15:34.807 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:34.796363 I | operator: successfully started the controller-runtime manager 2022-03-20T15:15:34.814 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:34.811147 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-20T15:15:34.814 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:34.811166 I | operator: watching all namespaces for Ceph CRs 2022-03-20T15:15:34.814 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:34.811190 I | operator: setting up schemes 2022-03-20T15:15:34.817 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:34.814230 I | operator: setting up the controller-runtime manager 2022-03-20T15:15:35.121 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.118567 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-20T15:15:35.156 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.152047 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-20T15:15:35.156 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.152083 I | ceph-cluster-controller: successfully started 2022-03-20T15:15:35.156 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.152142 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-20T15:15:35.156 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.152168 I | ceph-crashcollector-controller: successfully started 2022-03-20T15:15:35.158 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.152208 I | ceph-block-pool-controller: successfully started 2022-03-20T15:15:35.158 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.152241 I | ceph-object-store-user-controller: successfully started 2022-03-20T15:15:35.158 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.152269 I | ceph-object-realm-controller: successfully started 2022-03-20T15:15:35.158 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.152282 I | ceph-object-zonegroup-controller: successfully started 2022-03-20T15:15:35.159 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.152293 I | ceph-object-zone-controller: successfully started 2022-03-20T15:15:35.159 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.152436 I | ceph-object-controller: successfully started 2022-03-20T15:15:35.159 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.152484 I | ceph-file-controller: successfully started 2022-03-20T15:15:35.159 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.152538 I | ceph-nfs-controller: successfully started 2022-03-20T15:15:35.160 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.152583 I | ceph-rbd-mirror-controller: successfully started 2022-03-20T15:15:35.160 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.152628 I | ceph-client-controller: successfully started 2022-03-20T15:15:35.160 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.152663 I | ceph-filesystem-mirror-controller: successfully started 2022-03-20T15:15:35.160 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.152710 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-20T15:15:35.161 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.152741 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-20T15:15:35.161 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.152757 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-20T15:15:35.161 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.152777 I | ceph-bucket-topic: successfully started 2022-03-20T15:15:35.162 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.152789 I | ceph-bucket-notification: successfully started 2022-03-20T15:15:35.162 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.152802 I | ceph-bucket-notification: successfully started 2022-03-20T15:15:35.162 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.152813 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-20T15:15:35.162 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.154232 I | operator: starting the controller-runtime manager 2022-03-20T15:15:35.260 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.257106 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-20T15:15:35.260 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.257144 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-20T15:15:35.261 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.257155 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-20T15:15:35.265 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.258873 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-20T15:15:35.289 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.286786 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-20T15:15:35.323 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.320565 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:15:35.324 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.320659 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-20T15:15:35.324 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.320679 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-20T15:15:35.327 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.322563 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-20T15:15:35.327 INFO:tasks.rook.operator.smithi006.stdout:I0320 15:15:35.322690 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-20T15:15:35.330 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.324837 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:15:35.361 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.358505 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-20T15:15:35.366 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.360338 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e... 2022-03-20T15:15:35.454 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.449579 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-20T15:15:35.454 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.449612 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-20T15:15:35.454 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.449620 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-20T15:15:35.454 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.449627 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-20T15:15:35.455 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.449637 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-20T15:15:35.455 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.449649 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-20T15:15:35.455 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.449660 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-20T15:15:35.455 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.449668 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-20T15:15:35.456 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.449677 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-20T15:15:35.456 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.449684 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-20T15:15:35.456 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.449691 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-20T15:15:35.456 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.449701 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-20T15:15:35.456 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.449716 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-20T15:15:35.457 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.449726 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-20T15:15:35.457 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.449751 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-20T15:15:35.457 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.449764 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-20T15:15:35.457 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.449896 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-20T15:15:35.458 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:35.449912 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-20T15:15:39.271 INFO:teuthology.orchestra.run.smithi006.stdout:Success 2022-03-20T15:15:39.299 INFO:teuthology.orchestra.run.smithi006.stderr:+ removed_pv= 2022-03-20T15:15:39.299 INFO:teuthology.orchestra.run.smithi006.stderr:+ '[' '' = '' ']' 2022-03-20T15:15:39.300 INFO:teuthology.orchestra.run.smithi006.stderr:++ kubectl get pv 2022-03-20T15:15:39.300 INFO:teuthology.orchestra.run.smithi006.stderr:++ grep Released 2022-03-20T15:15:39.300 INFO:teuthology.orchestra.run.smithi006.stderr:++ cut -f1 -d ' ' 2022-03-20T15:15:39.378 INFO:teuthology.orchestra.run.smithi006.stderr:+ removed_pv= 2022-03-20T15:15:39.378 INFO:teuthology.orchestra.run.smithi006.stderr:+ sleep 3s 2022-03-20T15:15:40.801 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:40.797949 I | ceph-spec: object "rook-ceph-osd-0" matched on delete, reconciling 2022-03-20T15:15:42.379 INFO:teuthology.orchestra.run.smithi006.stderr:+ '[' '' = '' ']' 2022-03-20T15:15:42.380 INFO:teuthology.orchestra.run.smithi006.stderr:++ kubectl get pv 2022-03-20T15:15:42.380 INFO:teuthology.orchestra.run.smithi006.stderr:++ grep Released 2022-03-20T15:15:42.381 INFO:teuthology.orchestra.run.smithi006.stderr:++ cut -f1 -d ' ' 2022-03-20T15:15:42.451 INFO:teuthology.orchestra.run.smithi006.stderr:+ removed_pv=smithi045-lv-1 2022-03-20T15:15:42.452 INFO:teuthology.orchestra.run.smithi006.stderr:+ sleep 3s 2022-03-20T15:15:45.397 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.392284 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-20T15:15:45.397 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.392326 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-20T15:15:45.398 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.392339 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-20T15:15:45.398 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.392350 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-20T15:15:45.398 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.392361 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-20T15:15:45.398 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.392387 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-20T15:15:45.399 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.392398 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-20T15:15:45.399 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.392404 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-20T15:15:45.399 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.392412 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-20T15:15:45.399 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.392419 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-20T15:15:45.400 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.392428 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-20T15:15:45.400 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.392439 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-20T15:15:45.400 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.392450 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-20T15:15:45.400 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.392461 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-20T15:15:45.400 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.392472 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-20T15:15:45.401 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.392483 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-20T15:15:45.401 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.392493 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-20T15:15:45.401 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.392503 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-20T15:15:45.401 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.392514 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-20T15:15:45.402 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.392525 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-20T15:15:45.402 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.392536 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-20T15:15:45.402 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.392548 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-20T15:15:45.403 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.392562 I | ceph-csi: Kubernetes version is 1.23 2022-03-20T15:15:45.403 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.392576 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-20T15:15:45.403 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.392587 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-20T15:15:45.449 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.446265 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-20T15:15:45.452 INFO:teuthology.orchestra.run.smithi006.stderr:+ '[' smithi045-lv-1 = '' ']' 2022-03-20T15:15:45.453 INFO:teuthology.orchestra.run.smithi006.stderr:++ kubectl get pv smithi045-lv-1 -o 'jsonpath={.spec.local.path}' 2022-03-20T15:15:45.460 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.456698 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-20T15:15:45.460 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.456718 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-20T15:15:45.460 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.456724 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-20T15:15:45.461 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.456729 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-20T15:15:45.461 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.456735 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-20T15:15:45.461 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.456739 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-20T15:15:45.461 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.456745 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-20T15:15:45.549 INFO:teuthology.orchestra.run.smithi006.stderr:+ target_path=/dev/vg_nvme/lv_1 2022-03-20T15:15:45.549 INFO:teuthology.orchestra.run.smithi006.stderr:++ echo smithi045-lv-1 2022-03-20T15:15:45.550 INFO:teuthology.orchestra.run.smithi006.stderr:++ cut -f1 -d - 2022-03-20T15:15:45.550 INFO:teuthology.orchestra.run.smithi006.stderr:+ host=smithi045 2022-03-20T15:15:45.551 INFO:teuthology.orchestra.run.smithi006.stderr:+ toolbox ceph orch device zap smithi045 /dev/vg_nvme/lv_1 --force 2022-03-20T15:15:45.551 INFO:teuthology.orchestra.run.smithi006.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch device zap smithi045 /dev/vg_nvme/lv_1 --force 2022-03-20T15:15:45.594 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.590475 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-20T15:15:45.594 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.590506 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-20T15:15:45.595 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.590514 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-20T15:15:45.644 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.641540 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-20T15:15:45.645 INFO:teuthology.orchestra.run.smithi006.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-20T15:15:45.794 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.791465 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-20T15:15:45.795 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.791488 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-20T15:15:45.795 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.791495 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-20T15:15:45.892 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.889217 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-20T15:15:45.892 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.889239 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-20T15:15:45.893 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.889248 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-20T15:15:45.947 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:45.942362 I | ceph-csi: successfully started CSI CephFS driver 2022-03-20T15:15:46.096 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:46.093478 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-20T15:15:46.171 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:46.168555 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-20T15:15:46.172 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:46.168585 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-20T15:15:46.179 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:46.176681 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-20T15:15:46.261 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/vg_nvme/lv_1 on smithi045 zapped 2022-03-20T15:15:46.284 INFO:teuthology.orchestra.run.smithi006.stderr:+ zap_completion=0 2022-03-20T15:15:46.285 INFO:teuthology.orchestra.run.smithi006.stderr:+ '[' 0 = 0 ']' 2022-03-20T15:15:46.285 INFO:teuthology.orchestra.run.smithi006.stderr:++ kubectl get job -n rook-ceph rook-ceph-device-zap -o 'jsonpath={.status.succeeded.path}' 2022-03-20T15:15:46.393 INFO:teuthology.orchestra.run.smithi006.stderr:+ zap_completion= 2022-03-20T15:15:46.393 INFO:teuthology.orchestra.run.smithi006.stderr:+ sleep 3s 2022-03-20T15:15:47.251 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:47.247865 I | ceph-spec: detected ceph image version: "17.0.0-11187 quincy" 2022-03-20T15:15:47.251 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:47.247892 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-20T15:15:47.307 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:47.304467 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:15:47.313 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:47.310753 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T15:15:47.317 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:47.311086 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T15:15:47.738 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:47.734968 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11187 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e" 2022-03-20T15:15:47.868 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:47.864939 I | op-mon: start running mons 2022-03-20T15:15:47.892 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:47.889060 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:15:47.917 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:47.913956 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.99.78.235:6789","10.107.211.141:6789","10.97.13.112:6789"]}] data:b=10.97.13.112:6789,c=10.99.78.235:6789,a=10.107.211.141:6789 mapping:{"node":{"a":{"Name":"smithi045","Hostname":"smithi045","Address":"172.21.15.45"},"b":{"Name":"smithi123","Hostname":"smithi123","Address":"172.21.15.123"},"c":{"Name":"smithi006","Hostname":"smithi006","Address":"172.21.15.6"}}} maxMonId:2] 2022-03-20T15:15:48.396 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:48.393105 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T15:15:48.396 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:48.393329 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T15:15:49.394 INFO:teuthology.orchestra.run.smithi006.stderr:+ '[' '' = 0 ']' 2022-03-20T15:15:49.395 INFO:teuthology.orchestra.run.smithi006.stderr:+ kubectl patch pv smithi045-lv-1 -p '{"spec":{"claimRef": null}}' 2022-03-20T15:15:49.457 INFO:teuthology.orchestra.run.smithi006.stdout:persistentvolume/smithi045-lv-1 patched 2022-03-20T15:15:49.461 INFO:teuthology.orchestra.run.smithi006.stderr:+ toolbox ceph orch apply osd --all-available-devices 2022-03-20T15:15:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch apply osd --all-available-devices 2022-03-20T15:15:49.543 INFO:teuthology.orchestra.run.smithi006.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-20T15:15:50.009 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:50.006197 I | op-mon: targeting the mon count 3 2022-03-20T15:15:50.016 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:50.014061 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T15:15:50.856 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:50.853171 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T15:15:50.857 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:50.853209 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T15:15:50.899 INFO:teuthology.orchestra.run.smithi006.stdout:['Success'] 2022-03-20T15:15:50.910 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:50.905619 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-20T15:15:50.911 INFO:tasks.rook.operator.smithi006.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", AllowUnsupported: true}, 2022-03-20T15:15:50.911 INFO:tasks.rook.operator.smithi006.stdout: Storage: v1.StorageScopeSpec{ 2022-03-20T15:15:50.911 INFO:tasks.rook.operator.smithi006.stdout: ... // 3 identical fields 2022-03-20T15:15:50.911 INFO:tasks.rook.operator.smithi006.stdout: Config: nil, 2022-03-20T15:15:50.912 INFO:tasks.rook.operator.smithi006.stdout: Selection: {}, 2022-03-20T15:15:50.912 INFO:tasks.rook.operator.smithi006.stdout: StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-20T15:15:50.912 INFO:tasks.rook.operator.smithi006.stdout: ... // 9 identical elements 2022-03-20T15:15:50.912 INFO:tasks.rook.operator.smithi006.stdout: {Name: "smithi123-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi123-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-20T15:15:50.913 INFO:tasks.rook.operator.smithi006.stdout: {Name: "smithi123-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi123-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-03-20T15:15:50.913 INFO:tasks.rook.operator.smithi006.stdout:+ { 2022-03-20T15:15:50.913 INFO:tasks.rook.operator.smithi006.stdout:+ Name: "smithi045-lv-1", 2022-03-20T15:15:50.914 INFO:tasks.rook.operator.smithi006.stdout:+ Count: 1, 2022-03-20T15:15:50.914 INFO:tasks.rook.operator.smithi006.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-20T15:15:50.914 INFO:tasks.rook.operator.smithi006.stdout:+ }, 2022-03-20T15:15:50.914 INFO:tasks.rook.operator.smithi006.stdout: }, 2022-03-20T15:15:50.915 INFO:tasks.rook.operator.smithi006.stdout: }, 2022-03-20T15:15:50.915 INFO:tasks.rook.operator.smithi006.stdout: Annotations: nil, 2022-03-20T15:15:50.915 INFO:tasks.rook.operator.smithi006.stdout: Labels: nil, 2022-03-20T15:15:50.915 INFO:tasks.rook.operator.smithi006.stdout: ... // 20 identical fields 2022-03-20T15:15:50.916 INFO:tasks.rook.operator.smithi006.stdout: } 2022-03-20T15:15:50.916 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:50.905741 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-20T15:15:50.916 INFO:tasks.rook.operator.smithi006.stdout:I0320 15:15:50.906091 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-20T15:15:50.919 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:50.916026 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-20T15:15:50.919 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:50.916074 I | operator: watching all namespaces for Ceph CRs 2022-03-20T15:15:50.919 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:50.916117 I | operator: setting up schemes 2022-03-20T15:15:50.921 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:50.918626 I | operator: setting up the controller-runtime manager 2022-03-20T15:15:50.929 INFO:teuthology.orchestra.run.smithi006.stderr:+ kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-20T15:15:51.022 INFO:teuthology.orchestra.run.smithi006.stdout:job.batch "rook-ceph-device-zap" deleted 2022-03-20T15:15:51.028 INFO:teuthology.orchestra.run.smithi006.stderr:+ num_osd=0 2022-03-20T15:15:51.028 INFO:teuthology.orchestra.run.smithi006.stderr:+ '[' 0 '!=' 12 ']' 2022-03-20T15:15:51.028 INFO:teuthology.orchestra.run.smithi006.stderr:+ echo 'waiting for osd to come back up' 2022-03-20T15:15:51.029 INFO:teuthology.orchestra.run.smithi006.stdout:waiting for osd to come back up 2022-03-20T15:15:51.029 INFO:teuthology.orchestra.run.smithi006.stderr:++ toolbox ceph osd stat 2022-03-20T15:15:51.029 INFO:teuthology.orchestra.run.smithi006.stderr:++ cut -f3 -d ' ' 2022-03-20T15:15:51.030 INFO:teuthology.orchestra.run.smithi006.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-20T15:15:51.204 INFO:teuthology.orchestra.run.smithi006.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-20T15:15:51.226 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.222855 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-20T15:15:51.232 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.227030 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-20T15:15:51.232 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.227066 I | ceph-cluster-controller: successfully started 2022-03-20T15:15:51.232 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.227133 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-20T15:15:51.232 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.227167 I | ceph-crashcollector-controller: successfully started 2022-03-20T15:15:51.233 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.227207 I | ceph-block-pool-controller: successfully started 2022-03-20T15:15:51.233 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.227245 I | ceph-object-store-user-controller: successfully started 2022-03-20T15:15:51.233 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.227277 I | ceph-object-realm-controller: successfully started 2022-03-20T15:15:51.233 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.227301 I | ceph-object-zonegroup-controller: successfully started 2022-03-20T15:15:51.233 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.227329 I | ceph-object-zone-controller: successfully started 2022-03-20T15:15:51.234 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.227545 I | ceph-object-controller: successfully started 2022-03-20T15:15:51.234 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.227623 I | ceph-file-controller: successfully started 2022-03-20T15:15:51.234 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.227688 I | ceph-nfs-controller: successfully started 2022-03-20T15:15:51.234 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.227745 I | ceph-rbd-mirror-controller: successfully started 2022-03-20T15:15:51.235 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.227799 I | ceph-client-controller: successfully started 2022-03-20T15:15:51.235 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.227848 I | ceph-filesystem-mirror-controller: successfully started 2022-03-20T15:15:51.235 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.227905 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-20T15:15:51.235 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.227944 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-20T15:15:51.235 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.227981 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-20T15:15:51.236 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.228021 I | ceph-bucket-topic: successfully started 2022-03-20T15:15:51.236 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.228052 I | ceph-bucket-notification: successfully started 2022-03-20T15:15:51.236 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.228080 I | ceph-bucket-notification: successfully started 2022-03-20T15:15:51.236 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.228103 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-20T15:15:51.237 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.229591 I | operator: starting the controller-runtime manager 2022-03-20T15:15:51.338 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.334272 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-20T15:15:51.338 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.334344 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-20T15:15:51.338 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.334367 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-20T15:15:51.339 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.334562 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-20T15:15:51.345 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.340755 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-20T15:15:51.354 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.351605 I | op-mon: parsing mon endpoints: b=10.97.13.112:6789,c=10.99.78.235:6789,a=10.107.211.141:6789 2022-03-20T15:15:51.378 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.375096 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e... 2022-03-20T15:15:51.395 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.392719 I | op-mon: parsing mon endpoints: b=10.97.13.112:6789,c=10.99.78.235:6789,a=10.107.211.141:6789 2022-03-20T15:15:51.396 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.392818 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-20T15:15:51.396 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.392841 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-20T15:15:51.404 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.397413 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-20T15:15:51.404 INFO:tasks.rook.operator.smithi006.stdout:I0320 15:15:51.397582 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-20T15:15:51.605 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.602005 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-20T15:15:51.652 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.649734 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T15:15:51.653 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.649757 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T15:15:51.653 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.649812 W | op-mon: failed to set Rook and/or user-defined Ceph config options before starting mons; will retry after starting mons. failed to apply default Ceph configurations: failed to set one or more Ceph configs: failed to set ceph config in the centralized mon configuration database; you may need to use the rook-config-override ConfigMap. output: : context canceled 2022-03-20T15:15:51.654 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.649838 I | op-mon: checking for basic quorum with existing mons 2022-03-20T15:15:51.665 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.662906 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph monitors: failed to init mon services: context canceled 2022-03-20T15:15:51.669 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.663111 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-20T15:15:51.797 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.793891 I | op-k8sutil: Removing previous job rook-ceph-detect-version to start a new one 2022-03-20T15:15:51.802 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.799910 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-20T15:15:51.854 INFO:teuthology.orchestra.run.smithi006.stderr:+ num_osd=11 2022-03-20T15:15:51.854 INFO:teuthology.orchestra.run.smithi006.stderr:+ sleep 30s 2022-03-20T15:15:51.997 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.992798 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-20T15:15:51.998 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.992830 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-20T15:15:51.998 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.992839 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-20T15:15:51.998 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.992846 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-20T15:15:51.998 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.992854 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-20T15:15:51.999 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.992862 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-20T15:15:51.999 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.992871 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-20T15:15:51.999 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.992882 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-20T15:15:51.999 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.992895 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-20T15:15:52.000 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.992907 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-20T15:15:52.000 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.992922 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-20T15:15:52.000 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.992935 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-20T15:15:52.000 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.992954 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-20T15:15:52.000 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.992964 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-20T15:15:52.001 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.992974 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-20T15:15:52.001 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.992987 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-20T15:15:52.001 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.993079 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-20T15:15:52.001 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:51.993101 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-20T15:15:52.627 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:52.624192 I | op-mon: parsing mon endpoints: b=10.97.13.112:6789,c=10.99.78.235:6789,a=10.107.211.141:6789 2022-03-20T15:15:52.647 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:52.644743 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Detecting Ceph version LastHeartbeatTime:2022-03-20 15:15:52.634154096 +0000 UTC m=+536.867371257 LastTransitionTime:2022-03-20 15:15:52.634153913 +0000 UTC m=+536.867371120}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:15:52.648 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:52.644763 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e... 2022-03-20T15:15:54.809 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:54.806646 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-20T15:15:58.002 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:57.999536 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-20T15:15:58.835 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:58.832720 I | ceph-spec: detected ceph image version: "17.0.0-11187 quincy" 2022-03-20T15:15:58.836 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:58.832744 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-20T15:15:58.888 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:58.885718 I | op-mon: parsing mon endpoints: b=10.97.13.112:6789,c=10.99.78.235:6789,a=10.107.211.141:6789 2022-03-20T15:15:59.053 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.049742 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T15:15:59.104 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.095690 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T15:15:59.311 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.306603 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-20T15:15:59.312 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.306624 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-20T15:15:59.312 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.306631 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-20T15:15:59.312 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.306635 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-20T15:15:59.312 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.306637 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-20T15:15:59.313 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.306640 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-20T15:15:59.313 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.306643 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-20T15:15:59.313 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.306646 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-20T15:15:59.313 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.306651 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-20T15:15:59.314 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.306655 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-20T15:15:59.314 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.306659 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-20T15:15:59.314 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.306663 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-20T15:15:59.314 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.306667 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-20T15:15:59.314 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.306673 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-20T15:15:59.315 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.306677 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-20T15:15:59.315 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.306681 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-20T15:15:59.315 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.306686 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-20T15:15:59.315 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.306690 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-20T15:15:59.316 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.306696 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-20T15:15:59.316 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.306701 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-20T15:15:59.316 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.306707 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-20T15:15:59.316 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.306712 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-20T15:15:59.316 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.306717 I | ceph-csi: Kubernetes version is 1.23 2022-03-20T15:15:59.317 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.306723 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-20T15:15:59.317 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.306728 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-20T15:15:59.375 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.372242 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-20T15:15:59.380 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.377263 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-20T15:15:59.380 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.377277 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-20T15:15:59.381 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.377281 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-20T15:15:59.381 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.377283 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-20T15:15:59.381 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.377286 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-20T15:15:59.381 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.377288 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-20T15:15:59.382 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.377292 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-20T15:15:59.524 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.521449 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11187 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e" 2022-03-20T15:15:59.559 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.556047 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-20T15:15:59.559 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.556069 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-20T15:15:59.559 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.556080 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-20T15:15:59.593 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.590055 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring the Ceph cluster LastHeartbeatTime:2022-03-20 15:15:59.563708733 +0000 UTC m=+543.796925893 LastTransitionTime:2022-03-20 15:15:59.563708589 +0000 UTC m=+543.796925775}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:15:59.606 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.603182 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-20T15:15:59.685 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.682250 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring Ceph Mons LastHeartbeatTime:2022-03-20 15:15:59.602068123 +0000 UTC m=+543.835285275 LastTransitionTime:2022-03-20 15:15:59.602067953 +0000 UTC m=+543.835285129}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:15:59.685 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.682277 I | op-mon: start running mons 2022-03-20T15:15:59.713 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:15:59.710700 I | op-mon: parsing mon endpoints: b=10.97.13.112:6789,c=10.99.78.235:6789,a=10.107.211.141:6789 2022-03-20T15:16:00.170 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:00.167473 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-20T15:16:00.171 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:00.167498 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-20T15:16:00.171 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:00.167506 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-20T15:16:00.220 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:00.217607 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-20T15:16:00.221 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:00.217634 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-20T15:16:00.221 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:00.217641 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-20T15:16:00.268 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:00.266060 I | ceph-csi: successfully started CSI CephFS driver 2022-03-20T15:16:00.842 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:00.790566 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.13.112:6789","10.99.78.235:6789","10.107.211.141:6789"]}] data:b=10.97.13.112:6789,c=10.99.78.235:6789,a=10.107.211.141:6789 mapping:{"node":{"a":{"Name":"smithi045","Hostname":"smithi045","Address":"172.21.15.45"},"b":{"Name":"smithi123","Hostname":"smithi123","Address":"172.21.15.123"},"c":{"Name":"smithi006","Hostname":"smithi006","Address":"172.21.15.6"}}} maxMonId:2] 2022-03-20T15:16:01.006 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:01.003491 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-20T15:16:01.370 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:01.366891 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-20T15:16:01.375 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:01.372722 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-20T15:16:01.375 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:01.372757 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-20T15:16:01.383 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:01.377710 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-20T15:16:01.770 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:01.767643 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T15:16:01.774 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:01.767928 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T15:16:03.371 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:03.368584 I | op-mon: targeting the mon count 3 2022-03-20T15:16:03.377 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:03.374946 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T15:16:03.791 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:03.787500 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T15:16:03.791 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:03.787533 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T15:16:04.011 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:04.008399 I | op-k8sutil: batch job rook-ceph-detect-version deleted 2022-03-20T15:16:04.150 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:04.147264 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T15:16:04.151 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:04.147294 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T15:16:04.504 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:04.501343 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T15:16:04.504 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:04.501372 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-20T15:16:04.919 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:04.916057 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-20T15:16:04.920 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:04.916089 I | op-config: deleting "log file" option from the mon configuration database 2022-03-20T15:16:05.523 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:05.520009 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-20T15:16:05.523 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:05.520042 I | op-mon: checking for basic quorum with existing mons 2022-03-20T15:16:05.661 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:05.658688 I | op-mon: mon "b" ip is 10.97.13.112 2022-03-20T15:16:05.784 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:05.781677 I | op-mon: mon "c" ip is 10.99.78.235 2022-03-20T15:16:05.970 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:05.966981 I | op-mon: mon "a" ip is 10.107.211.141 2022-03-20T15:16:06.569 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:06.566855 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.13.112:6789","10.99.78.235:6789","10.107.211.141:6789"]}] data:b=10.97.13.112:6789,c=10.99.78.235:6789,a=10.107.211.141:6789 mapping:{"node":{"a":{"Name":"smithi045","Hostname":"smithi045","Address":"172.21.15.45"},"b":{"Name":"smithi123","Hostname":"smithi123","Address":"172.21.15.123"},"c":{"Name":"smithi006","Hostname":"smithi006","Address":"172.21.15.6"}}} maxMonId:2] 2022-03-20T15:16:07.170 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:07.167043 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T15:16:07.170 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:07.167212 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T15:16:07.613 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:07.609964 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-20T15:16:07.624 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:07.621098 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-20T15:16:07.624 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:07.621123 I | op-mon: waiting for mon quorum with [b c a] 2022-03-20T15:16:08.181 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:08.177879 I | op-mon: mons running: [b c a] 2022-03-20T15:16:08.604 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:08.601692 I | op-mon: Monitors in quorum: [a b c] 2022-03-20T15:16:08.628 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:08.625406 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-20T15:16:08.676 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:08.673805 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-20T15:16:08.677 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:08.673841 I | op-mon: waiting for mon quorum with [b c a] 2022-03-20T15:16:08.809 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:08.805650 I | op-mon: mons running: [b c a] 2022-03-20T15:16:09.264 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:09.260693 I | op-mon: Monitors in quorum: [a b c] 2022-03-20T15:16:09.272 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:09.269963 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-20T15:16:09.283 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:09.280063 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-20T15:16:09.283 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:09.280087 I | op-mon: waiting for mon quorum with [b c a] 2022-03-20T15:16:09.380 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:09.376864 I | op-mon: mons running: [b c a] 2022-03-20T15:16:09.806 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:09.802919 I | op-mon: Monitors in quorum: [a b c] 2022-03-20T15:16:09.806 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:09.802942 I | op-mon: mons created: 3 2022-03-20T15:16:10.092 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:10.089578 I | ceph-spec: detected ceph image version: "17.0.0-11187 quincy" 2022-03-20T15:16:10.093 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:10.089603 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-20T15:16:10.169 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:10.165977 I | op-mon: parsing mon endpoints: b=10.97.13.112:6789,c=10.99.78.235:6789,a=10.107.211.141:6789 2022-03-20T15:16:10.256 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:10.252853 I | op-mon: waiting for mon quorum with [b c a] 2022-03-20T15:16:10.371 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:10.368126 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T15:16:10.375 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:10.368406 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T15:16:10.783 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:10.780741 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11187 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e" 2022-03-20T15:16:10.812 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:10.809652 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring the Ceph cluster LastHeartbeatTime:2022-03-20 15:16:10.799761193 +0000 UTC m=+555.032978361 LastTransitionTime:2022-03-20 15:16:10.79976103 +0000 UTC m=+555.032978233}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:16:11.012 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:11.009450 I | op-mon: mons running: [b c a] 2022-03-20T15:16:11.205 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:11.201905 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring Ceph Mons LastHeartbeatTime:2022-03-20 15:16:11.191389623 +0000 UTC m=+555.424606805 LastTransitionTime:2022-03-20 15:16:11.19138946 +0000 UTC m=+555.424606660}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:16:11.427 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:11.423839 I | op-mon: Monitors in quorum: [a b c] 2022-03-20T15:16:11.427 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:11.423888 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-20T15:16:11.428 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:11.423973 I | op-mon: start running mons 2022-03-20T15:16:11.569 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:11.566586 I | op-mon: parsing mon endpoints: b=10.97.13.112:6789,c=10.99.78.235:6789,a=10.107.211.141:6789 2022-03-20T15:16:11.852 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:11.848833 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-20T15:16:12.401 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:12.397867 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-20T15:16:12.616 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:12.567853 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.13.112:6789","10.99.78.235:6789","10.107.211.141:6789"]}] data:b=10.97.13.112:6789,c=10.99.78.235:6789,a=10.107.211.141:6789 mapping:{"node":{"a":{"Name":"smithi045","Hostname":"smithi045","Address":"172.21.15.45"},"b":{"Name":"smithi123","Hostname":"smithi123","Address":"172.21.15.123"},"c":{"Name":"smithi006","Hostname":"smithi006","Address":"172.21.15.6"}}} maxMonId:2] 2022-03-20T15:16:12.828 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:12.825657 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-20T15:16:13.171 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:13.167898 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T15:16:13.171 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:13.168200 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T15:16:16.172 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:16.169537 I | op-mon: targeting the mon count 3 2022-03-20T15:16:16.180 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:16.177879 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T15:16:16.392 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:16.389584 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-20T15:16:16.393 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:16.389616 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-20T15:16:16.526 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:16.523818 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T15:16:16.529 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:16.523900 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T15:16:16.851 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:16.848062 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-20T15:16:16.851 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:16.848093 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-20T15:16:16.852 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:16.848102 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-20T15:16:16.873 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:16.870032 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T15:16:16.873 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:16.870063 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T15:16:17.220 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:17.217505 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T15:16:17.221 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:17.217545 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-20T15:16:17.269 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:17.266626 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-20T15:16:17.310 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:17.307475 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring Ceph Mgr(s) LastHeartbeatTime:2022-03-20 15:16:17.298330984 +0000 UTC m=+561.531548161 LastTransitionTime:2022-03-20 15:16:17.298330761 +0000 UTC m=+561.531547991}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:16:17.311 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:17.307500 I | op-mgr: start running mgr 2022-03-20T15:16:17.311 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:17.307516 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-20T15:16:17.585 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:17.581967 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-20T15:16:17.585 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:17.581998 I | op-config: deleting "log file" option from the mon configuration database 2022-03-20T15:16:17.806 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:17.802929 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-20T15:16:17.815 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:17.812525 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-20T15:16:17.819 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:17.815833 I | op-mgr: setting services to point to mgr "a" 2022-03-20T15:16:17.918 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:17.914828 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-20T15:16:17.918 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:17.914866 I | op-mon: checking for basic quorum with existing mons 2022-03-20T15:16:19.373 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:19.370441 I | op-mon: mon "c" ip is 10.99.78.235 2022-03-20T15:16:19.571 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:19.568175 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-20T15:16:19.572 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:19.568343 I | op-mgr: successful modules: balancer 2022-03-20T15:16:19.596 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:19.592740 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring Ceph OSDs LastHeartbeatTime:2022-03-20 15:16:19.568330721 +0000 UTC m=+563.801547918 LastTransitionTime:2022-03-20 15:16:19.568330434 +0000 UTC m=+563.801547695}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:16:19.597 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:19.592787 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-20T15:16:19.597 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:19.592802 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-20T15:16:19.641 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:19.638348 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-20T15:16:19.975 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:19.969721 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi006-lv-1" 2022-03-20T15:16:19.975 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:19.969775 I | op-osd: OSD PVC "smithi006-lv-1-data-0cmfzx" already exists 2022-03-20T15:16:19.975 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:19.969795 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi006-lv-2" 2022-03-20T15:16:19.976 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:19.969814 I | op-osd: OSD PVC "smithi006-lv-2-data-0hwbjq" already exists 2022-03-20T15:16:19.976 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:19.969900 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi006-lv-3" 2022-03-20T15:16:19.976 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:19.969945 I | op-osd: OSD PVC "smithi006-lv-3-data-07dd5g" already exists 2022-03-20T15:16:19.976 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:19.969978 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi006-lv-4" 2022-03-20T15:16:19.977 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:19.970002 I | op-osd: OSD PVC "smithi006-lv-4-data-0qbdz5" already exists 2022-03-20T15:16:19.977 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:19.970016 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi045-lv-2" 2022-03-20T15:16:19.977 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:19.970028 I | op-osd: OSD PVC "smithi045-lv-2-data-0lwhr2" already exists 2022-03-20T15:16:19.977 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:19.970045 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi045-lv-3" 2022-03-20T15:16:19.978 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:19.970071 I | op-osd: OSD PVC "smithi045-lv-3-data-0jqzrc" already exists 2022-03-20T15:16:19.978 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:19.970094 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi045-lv-4" 2022-03-20T15:16:19.978 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:19.970111 I | op-osd: OSD PVC "smithi045-lv-4-data-026444" already exists 2022-03-20T15:16:19.978 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:19.970119 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi123-lv-1" 2022-03-20T15:16:19.979 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:19.970131 I | op-osd: OSD PVC "smithi123-lv-1-data-0vz4f8" already exists 2022-03-20T15:16:19.979 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:19.970139 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi123-lv-2" 2022-03-20T15:16:19.979 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:19.970150 I | op-osd: OSD PVC "smithi123-lv-2-data-0l58zl" already exists 2022-03-20T15:16:19.979 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:19.970164 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi123-lv-3" 2022-03-20T15:16:19.980 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:19.970205 I | op-osd: OSD PVC "smithi123-lv-3-data-0crj7j" already exists 2022-03-20T15:16:19.980 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:19.970220 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi123-lv-4" 2022-03-20T15:16:19.980 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:19.970241 I | op-osd: OSD PVC "smithi123-lv-4-data-0vfnsc" already exists 2022-03-20T15:16:19.980 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:19.970253 I | op-osd: creating 1 new PVCs for device set "smithi045-lv-1" 2022-03-20T15:16:20.377 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:20.374124 I | op-osd: successfully provisioned PVC "smithi045-lv-1-data-0dqvrw" 2022-03-20T15:16:20.421 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:20.415289 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-1-data-0cmfzx" 2022-03-20T15:16:20.421 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:20.415311 I | op-osd: skipping OSD prepare job creation for PVC "smithi006-lv-1-data-0cmfzx" because OSD daemon using the PVC already exists 2022-03-20T15:16:20.421 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:20.415316 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-2-data-0hwbjq" 2022-03-20T15:16:20.421 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:20.415318 I | op-osd: skipping OSD prepare job creation for PVC "smithi006-lv-2-data-0hwbjq" because OSD daemon using the PVC already exists 2022-03-20T15:16:20.422 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:20.415322 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-3-data-07dd5g" 2022-03-20T15:16:20.422 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:20.415324 I | op-osd: skipping OSD prepare job creation for PVC "smithi006-lv-3-data-07dd5g" because OSD daemon using the PVC already exists 2022-03-20T15:16:20.422 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:20.415327 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-4-data-0qbdz5" 2022-03-20T15:16:20.422 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:20.415330 I | op-osd: skipping OSD prepare job creation for PVC "smithi006-lv-4-data-0qbdz5" because OSD daemon using the PVC already exists 2022-03-20T15:16:20.423 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:20.415333 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-2-data-0lwhr2" 2022-03-20T15:16:20.423 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:20.415336 I | op-osd: skipping OSD prepare job creation for PVC "smithi045-lv-2-data-0lwhr2" because OSD daemon using the PVC already exists 2022-03-20T15:16:20.423 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:20.415341 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-3-data-0jqzrc" 2022-03-20T15:16:20.423 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:20.415344 I | op-osd: skipping OSD prepare job creation for PVC "smithi045-lv-3-data-0jqzrc" because OSD daemon using the PVC already exists 2022-03-20T15:16:20.424 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:20.415346 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-4-data-026444" 2022-03-20T15:16:20.424 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:20.415349 I | op-osd: skipping OSD prepare job creation for PVC "smithi045-lv-4-data-026444" because OSD daemon using the PVC already exists 2022-03-20T15:16:20.424 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:20.415352 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-1-data-0vz4f8" 2022-03-20T15:16:20.424 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:20.415354 I | op-osd: skipping OSD prepare job creation for PVC "smithi123-lv-1-data-0vz4f8" because OSD daemon using the PVC already exists 2022-03-20T15:16:20.425 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:20.415357 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-2-data-0l58zl" 2022-03-20T15:16:20.425 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:20.415360 I | op-osd: skipping OSD prepare job creation for PVC "smithi123-lv-2-data-0l58zl" because OSD daemon using the PVC already exists 2022-03-20T15:16:20.425 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:20.415363 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-3-data-0crj7j" 2022-03-20T15:16:20.425 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:20.415365 I | op-osd: skipping OSD prepare job creation for PVC "smithi123-lv-3-data-0crj7j" because OSD daemon using the PVC already exists 2022-03-20T15:16:20.426 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:20.415370 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-4-data-0vfnsc" 2022-03-20T15:16:20.426 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:20.415373 I | op-osd: skipping OSD prepare job creation for PVC "smithi123-lv-4-data-0vfnsc" because OSD daemon using the PVC already exists 2022-03-20T15:16:20.426 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:20.415376 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-1-data-0dqvrw" 2022-03-20T15:16:20.930 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:20.927314 I | op-mgr: successful modules: dashboard 2022-03-20T15:16:20.932 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:20.927656 I | op-mgr: successful modules: prometheus 2022-03-20T15:16:20.933 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:20.928438 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-20T15:16:20.972 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:20.969705 I | op-mon: mon "a" ip is 10.107.211.141 2022-03-20T15:16:21.186 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:21.183355 I | op-osd: started OSD provisioning job for PVC "smithi045-lv-1-data-0dqvrw" 2022-03-20T15:16:21.187 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:21.183382 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-20T15:16:21.187 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:21.183389 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-20T15:16:21.579 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:21.576462 I | op-osd: OSD orchestration status for PVC smithi045-lv-1-data-0dqvrw is "starting" 2022-03-20T15:16:21.681 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:21.678315 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:16:21.686 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:21.683455 I | op-osd: updating OSD 1 on PVC "smithi006-lv-3-data-07dd5g" 2022-03-20T15:16:21.687 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:21.683493 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-3-data-07dd5g" 2022-03-20T15:16:21.703 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:21.700405 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 1 on PVC "smithi006-lv-3-data-07dd5g" LastHeartbeatTime:2022-03-20 15:16:21.693158108 +0000 UTC m=+565.926375278 LastTransitionTime:2022-03-20 15:16:21.693157918 +0000 UTC m=+565.926375134}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:16:21.825 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:21.822160 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:16:21.832 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:21.830036 I | op-osd: updating OSD 10 on PVC "smithi123-lv-3-data-0crj7j" 2022-03-20T15:16:21.833 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:21.830068 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-3-data-0crj7j" 2022-03-20T15:16:21.855 INFO:teuthology.orchestra.run.smithi006.stderr:+ '[' 11 '!=' 12 ']' 2022-03-20T15:16:21.856 INFO:teuthology.orchestra.run.smithi006.stderr:+ echo 'waiting for osd to come back up' 2022-03-20T15:16:21.856 INFO:teuthology.orchestra.run.smithi006.stdout:waiting for osd to come back up 2022-03-20T15:16:21.859 INFO:teuthology.orchestra.run.smithi006.stderr:++ toolbox ceph osd stat 2022-03-20T15:16:21.859 INFO:teuthology.orchestra.run.smithi006.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-20T15:16:21.859 INFO:teuthology.orchestra.run.smithi006.stderr:++ cut -f3 -d ' ' 2022-03-20T15:16:21.882 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:21.879518 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 10 on PVC "smithi123-lv-3-data-0crj7j" LastHeartbeatTime:2022-03-20 15:16:21.865053755 +0000 UTC m=+566.098270914 LastTransitionTime:2022-03-20 15:16:21.86505354 +0000 UTC m=+566.098270757}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:16:21.935 INFO:teuthology.orchestra.run.smithi006.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-20T15:16:22.028 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:22.024898 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:16:22.034 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:22.031678 I | op-osd: updating OSD 11 on PVC "smithi045-lv-3-data-0jqzrc" 2022-03-20T15:16:22.035 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:22.031705 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-3-data-0jqzrc" 2022-03-20T15:16:22.054 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:22.051305 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 11 on PVC "smithi045-lv-3-data-0jqzrc" LastHeartbeatTime:2022-03-20 15:16:22.042231706 +0000 UTC m=+566.275448865 LastTransitionTime:2022-03-20 15:16:22.042231581 +0000 UTC m=+566.275448747}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:16:22.192 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:22.189339 I | op-mon: mon "b" ip is 10.97.13.112 2022-03-20T15:16:22.196 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:22.192220 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:16:22.288 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:22.284840 I | op-osd: updating OSD 2 on PVC "smithi123-lv-1-data-0vz4f8" 2022-03-20T15:16:22.288 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:22.284882 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-1-data-0vz4f8" 2022-03-20T15:16:22.505 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:22.502155 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 2 on PVC "smithi123-lv-1-data-0vz4f8" LastHeartbeatTime:2022-03-20 15:16:22.492295248 +0000 UTC m=+566.725512438 LastTransitionTime:2022-03-20 15:16:22.492294981 +0000 UTC m=+566.725512215}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:16:22.723 INFO:teuthology.orchestra.run.smithi006.stderr:+ num_osd=11 2022-03-20T15:16:22.723 INFO:teuthology.orchestra.run.smithi006.stderr:+ sleep 30s 2022-03-20T15:16:22.771 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:22.768119 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.13.112:6789","10.99.78.235:6789","10.107.211.141:6789"]}] data:c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 mapping:{"node":{"a":{"Name":"smithi045","Hostname":"smithi045","Address":"172.21.15.45"},"b":{"Name":"smithi123","Hostname":"smithi123","Address":"172.21.15.123"},"c":{"Name":"smithi006","Hostname":"smithi006","Address":"172.21.15.6"}}} maxMonId:2] 2022-03-20T15:16:22.992 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:22.988867 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:16:23.077 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:23.074209 I | op-mgr: successful modules: orchestrator modules 2022-03-20T15:16:23.088 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:23.086065 I | op-osd: updating OSD 3 on PVC "smithi045-lv-2-data-0lwhr2" 2022-03-20T15:16:23.089 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:23.086091 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-2-data-0lwhr2" 2022-03-20T15:16:23.301 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:23.299023 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 3 on PVC "smithi045-lv-2-data-0lwhr2" LastHeartbeatTime:2022-03-20 15:16:23.291114652 +0000 UTC m=+567.524331843 LastTransitionTime:2022-03-20 15:16:23.291114382 +0000 UTC m=+567.524331620}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:16:23.369 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:23.366811 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T15:16:23.374 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:23.367100 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T15:16:23.837 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:23.834187 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:16:24.087 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:24.084637 I | op-osd: updating OSD 4 on PVC "smithi006-lv-2-data-0hwbjq" 2022-03-20T15:16:24.088 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:24.084661 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-2-data-0hwbjq" 2022-03-20T15:16:24.288 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:24.285957 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-20T15:16:24.511 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:24.508589 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 4 on PVC "smithi006-lv-2-data-0hwbjq" LastHeartbeatTime:2022-03-20 15:16:24.492142477 +0000 UTC m=+568.725359651 LastTransitionTime:2022-03-20 15:16:24.492142246 +0000 UTC m=+568.725359465}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:16:24.731 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:24.723774 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-20T15:16:24.731 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:24.723812 I | op-mon: waiting for mon quorum with [c a b] 2022-03-20T15:16:24.762 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:24.759718 I | op-mon: mons running: [c a b] 2022-03-20T15:16:25.173 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:25.170311 I | op-mon: Monitors in quorum: [a b c] 2022-03-20T15:16:25.195 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:25.192461 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:16:25.289 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:25.286068 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-20T15:16:25.487 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:25.484626 I | op-osd: updating OSD 5 on PVC "smithi006-lv-4-data-0qbdz5" 2022-03-20T15:16:25.487 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:25.484662 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-4-data-0qbdz5" 2022-03-20T15:16:25.692 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:25.689543 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-20T15:16:25.693 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:25.689573 I | op-mon: waiting for mon quorum with [c a b] 2022-03-20T15:16:25.730 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:25.727482 I | op-mon: mons running: [c a b] 2022-03-20T15:16:25.961 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:25.958317 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 5 on PVC "smithi006-lv-4-data-0qbdz5" LastHeartbeatTime:2022-03-20 15:16:25.892057731 +0000 UTC m=+570.125274910 LastTransitionTime:2022-03-20 15:16:25.892057499 +0000 UTC m=+570.125274735}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:16:26.164 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:26.161382 I | op-mon: Monitors in quorum: [a b c] 2022-03-20T15:16:26.395 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:26.392595 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:16:26.487 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:26.484648 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-20T15:16:26.690 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:26.686923 I | op-osd: updating OSD 6 on PVC "smithi006-lv-1-data-0cmfzx" 2022-03-20T15:16:26.690 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:26.686970 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-1-data-0cmfzx" 2022-03-20T15:16:26.901 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:26.898195 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-20T15:16:26.901 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:26.898230 I | op-mon: waiting for mon quorum with [c a b] 2022-03-20T15:16:27.071 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:27.068469 I | op-mon: mons running: [c a b] 2022-03-20T15:16:27.102 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:27.099573 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 6 on PVC "smithi006-lv-1-data-0cmfzx" LastHeartbeatTime:2022-03-20 15:16:27.090026436 +0000 UTC m=+571.323243589 LastTransitionTime:2022-03-20 15:16:27.090026315 +0000 UTC m=+571.323243482}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:16:27.490 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:27.487568 I | op-mon: Monitors in quorum: [a b c] 2022-03-20T15:16:27.490 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:27.487597 I | op-mon: mons created: 3 2022-03-20T15:16:27.595 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:27.592317 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:16:27.688 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:27.685336 I | op-osd: updating OSD 7 on PVC "smithi045-lv-4-data-026444" 2022-03-20T15:16:27.688 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:27.685375 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-4-data-026444" 2022-03-20T15:16:27.905 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:27.902295 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 7 on PVC "smithi045-lv-4-data-026444" LastHeartbeatTime:2022-03-20 15:16:27.892576756 +0000 UTC m=+572.125793950 LastTransitionTime:2022-03-20 15:16:27.892576513 +0000 UTC m=+572.125793741}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:16:27.916 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:27.913425 I | op-mon: waiting for mon quorum with [c a b] 2022-03-20T15:16:27.956 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:27.953332 I | op-mon: mons running: [c a b] 2022-03-20T15:16:28.366 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:28.363819 I | op-mon: Monitors in quorum: [a b c] 2022-03-20T15:16:28.367 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:28.363865 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-20T15:16:28.396 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:28.393375 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:16:28.488 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:28.484898 I | op-osd: updating OSD 8 on PVC "smithi123-lv-4-data-0vfnsc" 2022-03-20T15:16:28.488 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:28.484928 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-4-data-0vfnsc" 2022-03-20T15:16:28.714 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:28.711621 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 8 on PVC "smithi123-lv-4-data-0vfnsc" LastHeartbeatTime:2022-03-20 15:16:28.688874482 +0000 UTC m=+572.922091622 LastTransitionTime:2022-03-20 15:16:28.688874394 +0000 UTC m=+572.922091543}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:16:29.115 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:29.112608 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-20T15:16:29.197 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:29.195088 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:16:29.290 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:29.286839 I | op-osd: updating OSD 9 on PVC "smithi123-lv-2-data-0l58zl" 2022-03-20T15:16:29.290 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:29.286884 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-2-data-0l58zl" 2022-03-20T15:16:29.501 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:29.498229 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 9 on PVC "smithi123-lv-2-data-0l58zl" LastHeartbeatTime:2022-03-20 15:16:29.488539409 +0000 UTC m=+573.721756562 LastTransitionTime:2022-03-20 15:16:29.488539288 +0000 UTC m=+573.721756473}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:16:29.587 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:29.579744 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-20T15:16:30.059 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:30.056545 I | op-osd: OSD orchestration status for node smithi045-lv-1-data-0dqvrw is "orchestrating" 2022-03-20T15:16:30.177 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:30.174282 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-20T15:16:30.479 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:30.476266 I | op-osd: OSD orchestration status for PVC smithi045-lv-1-data-0dqvrw is "orchestrating" 2022-03-20T15:16:30.624 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:30.621567 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-20T15:16:30.625 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:30.621592 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-20T15:16:31.044 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:31.041088 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-20T15:16:31.044 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:31.041121 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-20T15:16:31.045 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:31.041130 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-20T15:16:31.483 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:31.480436 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-20T15:16:31.508 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:31.505727 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring Ceph Mgr(s) LastHeartbeatTime:2022-03-20 15:16:31.496844995 +0000 UTC m=+575.730062161 LastTransitionTime:2022-03-20 15:16:31.496844792 +0000 UTC m=+575.730061998}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:16:31.509 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:31.505747 I | op-mgr: start running mgr 2022-03-20T15:16:31.509 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:31.505763 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-20T15:16:32.135 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.132176 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-20T15:16:32.144 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.141701 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-20T15:16:32.148 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.145649 I | op-mgr: setting services to point to mgr "a" 2022-03-20T15:16:32.608 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.605089 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-20T15:16:32.608 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.605276 I | op-mgr: successful modules: balancer 2022-03-20T15:16:32.618 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.615857 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring Ceph OSDs LastHeartbeatTime:2022-03-20 15:16:32.605214329 +0000 UTC m=+576.838431520 LastTransitionTime:2022-03-20 15:16:32.605214055 +0000 UTC m=+576.838431315}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:16:32.619 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.615883 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-20T15:16:32.619 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.615891 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-20T15:16:32.651 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.648860 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-20T15:16:32.810 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.805959 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi006-lv-1" 2022-03-20T15:16:32.811 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.806004 I | op-osd: OSD PVC "smithi006-lv-1-data-0cmfzx" already exists 2022-03-20T15:16:32.811 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.806025 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi006-lv-2" 2022-03-20T15:16:32.811 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.806042 I | op-osd: OSD PVC "smithi006-lv-2-data-0hwbjq" already exists 2022-03-20T15:16:32.812 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.806052 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi006-lv-3" 2022-03-20T15:16:32.812 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.806066 I | op-osd: OSD PVC "smithi006-lv-3-data-07dd5g" already exists 2022-03-20T15:16:32.812 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.806085 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi006-lv-4" 2022-03-20T15:16:32.812 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.806111 I | op-osd: OSD PVC "smithi006-lv-4-data-0qbdz5" already exists 2022-03-20T15:16:32.813 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.806135 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi045-lv-2" 2022-03-20T15:16:32.813 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.806155 I | op-osd: OSD PVC "smithi045-lv-2-data-0lwhr2" already exists 2022-03-20T15:16:32.813 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.806171 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi045-lv-3" 2022-03-20T15:16:32.813 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.806189 I | op-osd: OSD PVC "smithi045-lv-3-data-0jqzrc" already exists 2022-03-20T15:16:32.813 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.806197 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi045-lv-4" 2022-03-20T15:16:32.814 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.806210 I | op-osd: OSD PVC "smithi045-lv-4-data-026444" already exists 2022-03-20T15:16:32.814 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.806222 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi123-lv-1" 2022-03-20T15:16:32.814 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.806238 I | op-osd: OSD PVC "smithi123-lv-1-data-0vz4f8" already exists 2022-03-20T15:16:32.814 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.806262 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi123-lv-2" 2022-03-20T15:16:32.815 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.806283 I | op-osd: OSD PVC "smithi123-lv-2-data-0l58zl" already exists 2022-03-20T15:16:32.815 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.806304 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi123-lv-3" 2022-03-20T15:16:32.815 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.806323 I | op-osd: OSD PVC "smithi123-lv-3-data-0crj7j" already exists 2022-03-20T15:16:32.815 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.806331 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi123-lv-4" 2022-03-20T15:16:32.816 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.806362 I | op-osd: OSD PVC "smithi123-lv-4-data-0vfnsc" already exists 2022-03-20T15:16:32.816 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.806373 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi045-lv-1" 2022-03-20T15:16:32.816 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.806384 I | op-osd: OSD PVC "smithi045-lv-1-data-0dqvrw" already exists 2022-03-20T15:16:32.838 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.833219 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-1-data-0cmfzx" 2022-03-20T15:16:32.838 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.833244 I | op-osd: skipping OSD prepare job creation for PVC "smithi006-lv-1-data-0cmfzx" because OSD daemon using the PVC already exists 2022-03-20T15:16:32.838 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.833251 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-2-data-0hwbjq" 2022-03-20T15:16:32.839 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.833257 I | op-osd: skipping OSD prepare job creation for PVC "smithi006-lv-2-data-0hwbjq" because OSD daemon using the PVC already exists 2022-03-20T15:16:32.839 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.833263 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-3-data-07dd5g" 2022-03-20T15:16:32.839 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.833268 I | op-osd: skipping OSD prepare job creation for PVC "smithi006-lv-3-data-07dd5g" because OSD daemon using the PVC already exists 2022-03-20T15:16:32.839 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.833272 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-4-data-0qbdz5" 2022-03-20T15:16:32.840 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.833277 I | op-osd: skipping OSD prepare job creation for PVC "smithi006-lv-4-data-0qbdz5" because OSD daemon using the PVC already exists 2022-03-20T15:16:32.840 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.833282 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-2-data-0lwhr2" 2022-03-20T15:16:32.840 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.833287 I | op-osd: skipping OSD prepare job creation for PVC "smithi045-lv-2-data-0lwhr2" because OSD daemon using the PVC already exists 2022-03-20T15:16:32.840 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.833292 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-3-data-0jqzrc" 2022-03-20T15:16:32.841 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.833297 I | op-osd: skipping OSD prepare job creation for PVC "smithi045-lv-3-data-0jqzrc" because OSD daemon using the PVC already exists 2022-03-20T15:16:32.841 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.833302 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-4-data-026444" 2022-03-20T15:16:32.841 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.833307 I | op-osd: skipping OSD prepare job creation for PVC "smithi045-lv-4-data-026444" because OSD daemon using the PVC already exists 2022-03-20T15:16:32.841 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.833314 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-1-data-0vz4f8" 2022-03-20T15:16:32.842 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.833319 I | op-osd: skipping OSD prepare job creation for PVC "smithi123-lv-1-data-0vz4f8" because OSD daemon using the PVC already exists 2022-03-20T15:16:32.842 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.833324 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-2-data-0l58zl" 2022-03-20T15:16:32.842 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.833329 I | op-osd: skipping OSD prepare job creation for PVC "smithi123-lv-2-data-0l58zl" because OSD daemon using the PVC already exists 2022-03-20T15:16:32.842 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.833334 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-3-data-0crj7j" 2022-03-20T15:16:32.842 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.833338 I | op-osd: skipping OSD prepare job creation for PVC "smithi123-lv-3-data-0crj7j" because OSD daemon using the PVC already exists 2022-03-20T15:16:32.843 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.833343 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-4-data-0vfnsc" 2022-03-20T15:16:32.843 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.833348 I | op-osd: skipping OSD prepare job creation for PVC "smithi123-lv-4-data-0vfnsc" because OSD daemon using the PVC already exists 2022-03-20T15:16:32.843 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:32.833353 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-1-data-0dqvrw" 2022-03-20T15:16:33.208 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:33.204896 I | op-osd: OSD orchestration status for PVC smithi045-lv-1-data-0dqvrw is "starting" 2022-03-20T15:16:33.214 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:33.211726 I | op-k8sutil: Found previous job rook-ceph-osd-prepare-smithi045-lv-1-data-0dqvrw. Status={Conditions:[] StartTime:2022-03-20 15:16:21 +0000 UTC CompletionTime: Active:1 Succeeded:0 Failed:0 CompletedIndexes: UncountedTerminatedPods:&UncountedTerminatedPods{Succeeded:[],Failed:[],}} 2022-03-20T15:16:33.215 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:33.211750 I | op-osd: started OSD provisioning job for PVC "smithi045-lv-1-data-0dqvrw" 2022-03-20T15:16:33.215 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:33.211759 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-20T15:16:33.215 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:33.211767 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-20T15:16:33.409 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:33.405991 I | op-osd: OSD orchestration status for PVC smithi045-lv-1-data-0dqvrw is "starting" 2022-03-20T15:16:33.511 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:33.507967 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:16:33.517 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:33.514240 I | op-osd: updating OSD 1 on PVC "smithi006-lv-3-data-07dd5g" 2022-03-20T15:16:33.517 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:33.514271 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-3-data-07dd5g" 2022-03-20T15:16:33.535 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:33.532833 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 1 on PVC "smithi006-lv-3-data-07dd5g" LastHeartbeatTime:2022-03-20 15:16:33.524532159 +0000 UTC m=+577.757749327 LastTransitionTime:2022-03-20 15:16:33.524531931 +0000 UTC m=+577.757749147}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:16:33.658 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:33.654024 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:16:33.673 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:33.670636 I | op-osd: updating OSD 10 on PVC "smithi123-lv-3-data-0crj7j" 2022-03-20T15:16:33.673 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:33.670662 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-3-data-0crj7j" 2022-03-20T15:16:33.749 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:33.746749 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 10 on PVC "smithi123-lv-3-data-0crj7j" LastHeartbeatTime:2022-03-20 15:16:33.736749189 +0000 UTC m=+577.969966407 LastTransitionTime:2022-03-20 15:16:33.736748956 +0000 UTC m=+577.969966160}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:16:33.819 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:33.815835 I | op-mgr: successful modules: prometheus 2022-03-20T15:16:33.842 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:33.839084 I | op-mgr: successful modules: dashboard 2022-03-20T15:16:33.845 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:33.841243 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-20T15:16:33.873 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:33.870815 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:16:33.880 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:33.877468 I | op-osd: updating OSD 11 on PVC "smithi045-lv-3-data-0jqzrc" 2022-03-20T15:16:33.880 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:33.877499 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-3-data-0jqzrc" 2022-03-20T15:16:33.898 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:33.895874 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 11 on PVC "smithi045-lv-3-data-0jqzrc" LastHeartbeatTime:2022-03-20 15:16:33.886889279 +0000 UTC m=+578.120106466 LastTransitionTime:2022-03-20 15:16:33.886889056 +0000 UTC m=+578.120106290}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:16:34.024 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:34.021240 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:16:34.119 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:34.115909 I | op-osd: updating OSD 2 on PVC "smithi123-lv-1-data-0vz4f8" 2022-03-20T15:16:34.119 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:34.115937 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-1-data-0vz4f8" 2022-03-20T15:16:34.331 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:34.328396 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 2 on PVC "smithi123-lv-1-data-0vz4f8" LastHeartbeatTime:2022-03-20 15:16:34.320272174 +0000 UTC m=+578.553489377 LastTransitionTime:2022-03-20 15:16:34.320271869 +0000 UTC m=+578.553489148}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:16:34.822 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:34.819613 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:16:34.919 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:34.915360 I | op-osd: updating OSD 3 on PVC "smithi045-lv-2-data-0lwhr2" 2022-03-20T15:16:34.919 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:34.915391 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-2-data-0lwhr2" 2022-03-20T15:16:35.134 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:35.131685 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 3 on PVC "smithi045-lv-2-data-0lwhr2" LastHeartbeatTime:2022-03-20 15:16:35.122064122 +0000 UTC m=+579.355281331 LastTransitionTime:2022-03-20 15:16:35.122063831 +0000 UTC m=+579.355281102}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:16:35.449 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:35.446707 I | op-mgr: successful modules: orchestrator modules 2022-03-20T15:16:35.627 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:35.624457 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:16:35.724 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:35.721583 I | op-osd: updating OSD 4 on PVC "smithi006-lv-2-data-0hwbjq" 2022-03-20T15:16:35.725 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:35.721616 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-2-data-0hwbjq" 2022-03-20T15:16:35.935 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:35.932570 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 4 on PVC "smithi006-lv-2-data-0hwbjq" LastHeartbeatTime:2022-03-20 15:16:35.921511413 +0000 UTC m=+580.154728618 LastTransitionTime:2022-03-20 15:16:35.921511167 +0000 UTC m=+580.154728390}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:16:36.423 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:36.420449 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:16:36.551 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:36.548228 I | op-osd: updating OSD 5 on PVC "smithi006-lv-4-data-0qbdz5" 2022-03-20T15:16:36.551 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:36.548269 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-4-data-0qbdz5" 2022-03-20T15:16:36.587 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:36.584834 I | op-osd: OSD orchestration status for PVC smithi045-lv-1-data-0dqvrw is "completed" 2022-03-20T15:16:36.588 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:36.584869 I | op-osd: creating OSD 0 on PVC "smithi045-lv-1-data-0dqvrw" 2022-03-20T15:16:36.588 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:36.584885 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-1-data-0dqvrw" 2022-03-20T15:16:36.733 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:36.730874 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 5 on PVC "smithi006-lv-4-data-0qbdz5" LastHeartbeatTime:2022-03-20 15:16:36.72147133 +0000 UTC m=+580.954688504 LastTransitionTime:2022-03-20 15:16:36.721471019 +0000 UTC m=+580.954688282}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:16:36.948 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:36.945500 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 0 on PVC "smithi045-lv-1-data-0dqvrw" LastHeartbeatTime:2022-03-20 15:16:36.936195684 +0000 UTC m=+581.169412848 LastTransitionTime:2022-03-20 15:16:36.936195528 +0000 UTC m=+581.169412714}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:16:37.522 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:37.519810 I | op-osd: OSD orchestration status for PVC smithi045-lv-1-data-0dqvrw is "completed" 2022-03-20T15:16:37.523 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:37.519833 I | op-osd: creating OSD 0 on PVC "smithi045-lv-1-data-0dqvrw" 2022-03-20T15:16:37.523 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:37.519844 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-1-data-0dqvrw" 2022-03-20T15:16:37.731 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:37.727943 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 0 on PVC "smithi045-lv-1-data-0dqvrw" LastHeartbeatTime:2022-03-20 15:16:37.720636657 +0000 UTC m=+581.953853799 LastTransitionTime:2022-03-20 15:16:37.720636534 +0000 UTC m=+581.953853705}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:16:37.865 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:37.862454 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-20T15:16:37.866 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:37.862484 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-20T15:16:37.887 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:37.884874 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-20 15:16:37.862581106 +0000 UTC m=+582.095798286 LastTransitionTime:2022-03-20 15:12:22 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:16:37.973 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:37.970499 E | op-osd: failed to create OSD 0 on PVC "smithi045-lv-1-data-0dqvrw": failed to create deployment for OSD 0 on PVC "smithi045-lv-1-data-0dqvrw": deployments.apps "rook-ceph-osd-0" already exists 2022-03-20T15:16:38.076 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:38.073676 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:16:38.120 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:38.116433 I | op-osd: updating OSD 6 on PVC "smithi006-lv-1-data-0cmfzx" 2022-03-20T15:16:38.120 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:38.116463 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-1-data-0cmfzx" 2022-03-20T15:16:38.335 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:38.332162 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 6 on PVC "smithi006-lv-1-data-0cmfzx" LastHeartbeatTime:2022-03-20 15:16:38.324202577 +0000 UTC m=+582.557419754 LastTransitionTime:2022-03-20 15:16:38.324202343 +0000 UTC m=+582.557419557}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:16:38.825 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:38.822701 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:16:38.919 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:38.916346 I | op-osd: updating OSD 7 on PVC "smithi045-lv-4-data-026444" 2022-03-20T15:16:38.919 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:38.916383 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-4-data-026444" 2022-03-20T15:16:39.152 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:39.149390 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 7 on PVC "smithi045-lv-4-data-026444" LastHeartbeatTime:2022-03-20 15:16:39.138255128 +0000 UTC m=+583.371472299 LastTransitionTime:2022-03-20 15:16:39.138254897 +0000 UTC m=+583.371472111}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:16:39.628 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:39.625291 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:16:39.718 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:39.715098 I | op-osd: updating OSD 8 on PVC "smithi123-lv-4-data-0vfnsc" 2022-03-20T15:16:39.718 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:39.715134 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-4-data-0vfnsc" 2022-03-20T15:16:39.935 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:39.932297 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 8 on PVC "smithi123-lv-4-data-0vfnsc" LastHeartbeatTime:2022-03-20 15:16:39.922711174 +0000 UTC m=+584.155928348 LastTransitionTime:2022-03-20 15:16:39.922710907 +0000 UTC m=+584.155928141}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:16:40.455 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:40.452117 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:16:40.519 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:40.515960 I | op-osd: updating OSD 9 on PVC "smithi123-lv-2-data-0l58zl" 2022-03-20T15:16:40.519 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:40.515996 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-2-data-0l58zl" 2022-03-20T15:16:40.732 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:40.729233 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 9 on PVC "smithi123-lv-2-data-0l58zl" LastHeartbeatTime:2022-03-20 15:16:40.721421727 +0000 UTC m=+584.954638921 LastTransitionTime:2022-03-20 15:16:40.721421516 +0000 UTC m=+584.954638726}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:16:41.134 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:41.130886 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:False Reason:ClusterProgressing Message:failed to create cluster: failed to start ceph osds: 1 failures encountered while running osds on nodes in namespace "rook-ceph". 2022-03-20T15:16:41.134 INFO:tasks.rook.operator.smithi006.stdout:failed to create OSD 0 on PVC "smithi045-lv-1-data-0dqvrw": failed to create deployment for OSD 0 on PVC "smithi045-lv-1-data-0dqvrw": deployments.apps "rook-ceph-osd-0" already exists LastHeartbeatTime:2022-03-20 15:16:41.123321992 +0000 UTC m=+585.356539160 LastTransitionTime:2022-03-20 15:16:41.123321829 +0000 UTC m=+585.356539017}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:16:41.134 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:41.130974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph osds: 1 failures encountered while running osds on nodes in namespace "rook-ceph". 2022-03-20T15:16:41.135 INFO:tasks.rook.operator.smithi006.stdout:failed to create OSD 0 on PVC "smithi045-lv-1-data-0dqvrw": failed to create deployment for OSD 0 on PVC "smithi045-lv-1-data-0dqvrw": deployments.apps "rook-ceph-osd-0" already exists 2022-03-20T15:16:41.143 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:41.137183 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-20T15:16:41.149 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:41.146936 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:16:41.367 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:41.364476 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e... 2022-03-20T15:16:49.990 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:49.987754 I | ceph-spec: detected ceph image version: "17.0.0-11187 quincy" 2022-03-20T15:16:49.991 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:49.987783 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-20T15:16:49.998 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:49.996176 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:16:50.021 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:50.019025 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T15:16:50.022 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:50.019207 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T15:16:50.525 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:50.522667 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11187 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e" 2022-03-20T15:16:50.594 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:50.591848 I | op-mon: start running mons 2022-03-20T15:16:50.617 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:50.614416 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:16:50.684 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:50.681323 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.99.78.235:6789","10.107.211.141:6789","10.97.13.112:6789"]}] data:c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 mapping:{"node":{"a":{"Name":"smithi045","Hostname":"smithi045","Address":"172.21.15.45"},"b":{"Name":"smithi123","Hostname":"smithi123","Address":"172.21.15.123"},"c":{"Name":"smithi006","Hostname":"smithi006","Address":"172.21.15.6"}}} maxMonId:2] 2022-03-20T15:16:51.072 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:51.069190 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T15:16:51.076 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:51.069492 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T15:16:52.649 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:52.646286 I | op-mon: targeting the mon count 3 2022-03-20T15:16:52.655 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:52.653248 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T15:16:52.724 INFO:teuthology.orchestra.run.smithi006.stderr:+ '[' 11 '!=' 12 ']' 2022-03-20T15:16:52.725 INFO:teuthology.orchestra.run.smithi006.stdout:waiting for osd to come back up 2022-03-20T15:16:52.726 INFO:teuthology.orchestra.run.smithi006.stderr:+ echo 'waiting for osd to come back up' 2022-03-20T15:16:52.726 INFO:teuthology.orchestra.run.smithi006.stderr:++ toolbox ceph osd stat 2022-03-20T15:16:52.726 INFO:teuthology.orchestra.run.smithi006.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-20T15:16:52.726 INFO:teuthology.orchestra.run.smithi006.stderr:++ cut -f3 -d ' ' 2022-03-20T15:16:52.880 INFO:teuthology.orchestra.run.smithi006.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-20T15:16:53.023 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:53.020251 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T15:16:53.023 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:53.020284 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T15:16:53.676 INFO:teuthology.orchestra.run.smithi006.stderr:+ num_osd=11 2022-03-20T15:16:53.677 INFO:teuthology.orchestra.run.smithi006.stderr:+ sleep 30s 2022-03-20T15:16:53.677 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:53.674035 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T15:16:53.678 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:53.674068 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T15:16:54.043 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:54.040627 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T15:16:54.044 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:54.040660 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-20T15:16:54.390 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:54.387442 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-20T15:16:54.391 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:54.387469 I | op-config: deleting "log file" option from the mon configuration database 2022-03-20T15:16:54.769 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:54.766917 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-20T15:16:54.770 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:54.766946 I | op-mon: checking for basic quorum with existing mons 2022-03-20T15:16:54.958 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:54.955439 I | op-mon: mon "b" ip is 10.97.13.112 2022-03-20T15:16:55.092 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:55.089440 I | op-mon: mon "c" ip is 10.99.78.235 2022-03-20T15:16:55.196 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:55.193725 I | op-mon: mon "a" ip is 10.107.211.141 2022-03-20T15:16:55.780 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:55.776565 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.99.78.235:6789","10.107.211.141:6789","10.97.13.112:6789"]}] data:c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 mapping:{"node":{"a":{"Name":"smithi045","Hostname":"smithi045","Address":"172.21.15.45"},"b":{"Name":"smithi123","Hostname":"smithi123","Address":"172.21.15.123"},"c":{"Name":"smithi006","Hostname":"smithi006","Address":"172.21.15.6"}}} maxMonId:2] 2022-03-20T15:16:56.375 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:56.372662 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T15:16:56.380 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:56.372971 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T15:16:56.791 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:56.788344 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-20T15:16:56.802 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:56.799313 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-20T15:16:56.802 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:56.799349 I | op-mon: waiting for mon quorum with [b c a] 2022-03-20T15:16:57.385 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:57.382188 I | op-mon: mons running: [b c a] 2022-03-20T15:16:57.802 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:57.799027 I | op-mon: Monitors in quorum: [a b c] 2022-03-20T15:16:57.809 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:57.807020 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-20T15:16:57.820 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:57.817713 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-20T15:16:57.820 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:57.817739 I | op-mon: waiting for mon quorum with [b c a] 2022-03-20T15:16:57.985 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:57.982267 I | op-mon: mons running: [b c a] 2022-03-20T15:16:58.429 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:58.426103 I | op-mon: Monitors in quorum: [a b c] 2022-03-20T15:16:58.438 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:58.435966 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-20T15:16:58.449 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:58.446422 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-20T15:16:58.449 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:58.446440 I | op-mon: waiting for mon quorum with [b c a] 2022-03-20T15:16:58.584 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:58.582041 I | op-mon: mons running: [b c a] 2022-03-20T15:16:59.035 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:59.032002 I | op-mon: Monitors in quorum: [a b c] 2022-03-20T15:16:59.035 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:59.032034 I | op-mon: mons created: 3 2022-03-20T15:16:59.475 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:59.471080 I | op-mon: waiting for mon quorum with [b c a] 2022-03-20T15:16:59.518 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:59.515872 I | op-mon: mons running: [b c a] 2022-03-20T15:16:59.938 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:59.934998 I | op-mon: Monitors in quorum: [a b c] 2022-03-20T15:16:59.938 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:16:59.935038 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-20T15:17:00.358 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:00.355414 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-20T15:17:00.841 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:00.838131 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-20T15:17:01.261 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:01.257983 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-20T15:17:01.824 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:01.821640 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-20T15:17:01.824 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:01.821670 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-20T15:17:02.271 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:02.268027 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-20T15:17:02.271 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:02.268072 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-20T15:17:02.271 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:02.268080 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-20T15:17:02.688 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:02.685250 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-20T15:17:02.738 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:02.735877 I | op-mgr: start running mgr 2022-03-20T15:17:02.739 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:02.735907 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-20T15:17:03.234 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:03.231949 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-20T15:17:03.245 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:03.242696 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-20T15:17:03.304 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:03.302166 I | op-mgr: setting services to point to mgr "a" 2022-03-20T15:17:03.798 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:03.795173 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-20T15:17:03.799 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:03.795344 I | op-mgr: successful modules: balancer 2022-03-20T15:17:03.859 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:03.855858 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-20T15:17:03.859 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:03.855897 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-20T15:17:03.887 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:03.884290 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-20T15:17:04.002 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:03.997171 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi006-lv-1" 2022-03-20T15:17:04.003 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:03.997230 I | op-osd: OSD PVC "smithi006-lv-1-data-0cmfzx" already exists 2022-03-20T15:17:04.003 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:03.997249 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi006-lv-2" 2022-03-20T15:17:04.004 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:03.997278 I | op-osd: OSD PVC "smithi006-lv-2-data-0hwbjq" already exists 2022-03-20T15:17:04.004 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:03.997297 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi006-lv-3" 2022-03-20T15:17:04.004 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:03.997318 I | op-osd: OSD PVC "smithi006-lv-3-data-07dd5g" already exists 2022-03-20T15:17:04.005 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:03.997337 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi006-lv-4" 2022-03-20T15:17:04.005 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:03.997357 I | op-osd: OSD PVC "smithi006-lv-4-data-0qbdz5" already exists 2022-03-20T15:17:04.005 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:03.997372 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi045-lv-2" 2022-03-20T15:17:04.006 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:03.997393 I | op-osd: OSD PVC "smithi045-lv-2-data-0lwhr2" already exists 2022-03-20T15:17:04.006 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:03.997413 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi045-lv-3" 2022-03-20T15:17:04.007 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:03.997437 I | op-osd: OSD PVC "smithi045-lv-3-data-0jqzrc" already exists 2022-03-20T15:17:04.007 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:03.997462 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi045-lv-4" 2022-03-20T15:17:04.007 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:03.997483 I | op-osd: OSD PVC "smithi045-lv-4-data-026444" already exists 2022-03-20T15:17:04.008 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:03.997506 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi123-lv-1" 2022-03-20T15:17:04.008 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:03.997528 I | op-osd: OSD PVC "smithi123-lv-1-data-0vz4f8" already exists 2022-03-20T15:17:04.008 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:03.997549 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi123-lv-2" 2022-03-20T15:17:04.009 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:03.997572 I | op-osd: OSD PVC "smithi123-lv-2-data-0l58zl" already exists 2022-03-20T15:17:04.009 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:03.997588 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi123-lv-3" 2022-03-20T15:17:04.009 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:03.997604 I | op-osd: OSD PVC "smithi123-lv-3-data-0crj7j" already exists 2022-03-20T15:17:04.010 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:03.997615 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi123-lv-4" 2022-03-20T15:17:04.010 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:03.997638 I | op-osd: OSD PVC "smithi123-lv-4-data-0vfnsc" already exists 2022-03-20T15:17:04.010 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:03.997654 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi045-lv-1" 2022-03-20T15:17:04.010 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:03.997674 I | op-osd: OSD PVC "smithi045-lv-1-data-0dqvrw" already exists 2022-03-20T15:17:04.033 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.028259 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-1-data-0cmfzx" 2022-03-20T15:17:04.033 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.028286 I | op-osd: skipping OSD prepare job creation for PVC "smithi006-lv-1-data-0cmfzx" because OSD daemon using the PVC already exists 2022-03-20T15:17:04.033 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.028294 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-2-data-0hwbjq" 2022-03-20T15:17:04.034 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.028299 I | op-osd: skipping OSD prepare job creation for PVC "smithi006-lv-2-data-0hwbjq" because OSD daemon using the PVC already exists 2022-03-20T15:17:04.034 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.028304 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-3-data-07dd5g" 2022-03-20T15:17:04.034 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.028309 I | op-osd: skipping OSD prepare job creation for PVC "smithi006-lv-3-data-07dd5g" because OSD daemon using the PVC already exists 2022-03-20T15:17:04.034 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.028313 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-4-data-0qbdz5" 2022-03-20T15:17:04.035 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.028318 I | op-osd: skipping OSD prepare job creation for PVC "smithi006-lv-4-data-0qbdz5" because OSD daemon using the PVC already exists 2022-03-20T15:17:04.035 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.028323 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-2-data-0lwhr2" 2022-03-20T15:17:04.035 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.028328 I | op-osd: skipping OSD prepare job creation for PVC "smithi045-lv-2-data-0lwhr2" because OSD daemon using the PVC already exists 2022-03-20T15:17:04.035 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.028332 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-3-data-0jqzrc" 2022-03-20T15:17:04.035 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.028337 I | op-osd: skipping OSD prepare job creation for PVC "smithi045-lv-3-data-0jqzrc" because OSD daemon using the PVC already exists 2022-03-20T15:17:04.036 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.028342 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-4-data-026444" 2022-03-20T15:17:04.036 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.028346 I | op-osd: skipping OSD prepare job creation for PVC "smithi045-lv-4-data-026444" because OSD daemon using the PVC already exists 2022-03-20T15:17:04.036 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.028351 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-1-data-0vz4f8" 2022-03-20T15:17:04.036 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.028356 I | op-osd: skipping OSD prepare job creation for PVC "smithi123-lv-1-data-0vz4f8" because OSD daemon using the PVC already exists 2022-03-20T15:17:04.037 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.028360 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-2-data-0l58zl" 2022-03-20T15:17:04.037 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.028365 I | op-osd: skipping OSD prepare job creation for PVC "smithi123-lv-2-data-0l58zl" because OSD daemon using the PVC already exists 2022-03-20T15:17:04.037 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.028370 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-3-data-0crj7j" 2022-03-20T15:17:04.037 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.028376 I | op-osd: skipping OSD prepare job creation for PVC "smithi123-lv-3-data-0crj7j" because OSD daemon using the PVC already exists 2022-03-20T15:17:04.037 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.028381 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-4-data-0vfnsc" 2022-03-20T15:17:04.038 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.028386 I | op-osd: skipping OSD prepare job creation for PVC "smithi123-lv-4-data-0vfnsc" because OSD daemon using the PVC already exists 2022-03-20T15:17:04.038 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.028391 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-1-data-0dqvrw" 2022-03-20T15:17:04.038 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.028395 I | op-osd: skipping OSD prepare job creation for PVC "smithi045-lv-1-data-0dqvrw" because OSD daemon using the PVC already exists 2022-03-20T15:17:04.039 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.028399 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-20T15:17:04.039 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.028404 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-20T15:17:04.301 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.297994 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:17:04.307 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.304764 I | op-osd: updating OSD 0 on PVC "smithi045-lv-1-data-0dqvrw" 2022-03-20T15:17:04.307 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.304790 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-1-data-0dqvrw" 2022-03-20T15:17:04.483 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.480789 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:17:04.490 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.487566 I | op-osd: updating OSD 1 on PVC "smithi006-lv-3-data-07dd5g" 2022-03-20T15:17:04.490 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.487598 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-3-data-07dd5g" 2022-03-20T15:17:04.682 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.679337 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:17:04.688 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.686070 I | op-osd: updating OSD 10 on PVC "smithi123-lv-3-data-0crj7j" 2022-03-20T15:17:04.688 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.686112 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-3-data-0crj7j" 2022-03-20T15:17:04.867 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.864367 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:17:04.907 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.904857 I | op-osd: updating OSD 11 on PVC "smithi045-lv-3-data-0jqzrc" 2022-03-20T15:17:04.907 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:04.904884 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-3-data-0jqzrc" 2022-03-20T15:17:05.410 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:05.407846 I | op-mgr: successful modules: prometheus 2022-03-20T15:17:05.441 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:05.438566 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-20T15:17:05.441 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:05.438759 I | op-mgr: successful modules: dashboard 2022-03-20T15:17:05.614 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:05.611114 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:17:05.708 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:05.705765 I | op-osd: updating OSD 2 on PVC "smithi123-lv-1-data-0vz4f8" 2022-03-20T15:17:05.708 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:05.705799 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-1-data-0vz4f8" 2022-03-20T15:17:06.416 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:06.413005 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:17:06.507 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:06.504585 I | op-osd: updating OSD 3 on PVC "smithi045-lv-2-data-0lwhr2" 2022-03-20T15:17:06.507 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:06.504616 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-2-data-0lwhr2" 2022-03-20T15:17:07.048 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:07.045353 I | op-mgr: successful modules: orchestrator modules 2022-03-20T15:17:07.221 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:07.212105 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:17:07.311 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:07.308855 I | op-osd: updating OSD 4 on PVC "smithi006-lv-2-data-0hwbjq" 2022-03-20T15:17:07.312 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:07.308892 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-2-data-0hwbjq" 2022-03-20T15:17:08.019 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:08.016459 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:17:08.107 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:08.104678 I | op-osd: updating OSD 5 on PVC "smithi006-lv-4-data-0qbdz5" 2022-03-20T15:17:08.107 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:08.104718 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-4-data-0qbdz5" 2022-03-20T15:17:08.817 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:08.813990 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:17:08.907 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:08.904156 I | op-osd: updating OSD 6 on PVC "smithi006-lv-1-data-0cmfzx" 2022-03-20T15:17:08.907 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:08.904181 I | op-osd: OSD will have its main bluestore block on "smithi006-lv-1-data-0cmfzx" 2022-03-20T15:17:09.615 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:09.612918 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:17:09.707 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:09.704197 I | op-osd: updating OSD 7 on PVC "smithi045-lv-4-data-026444" 2022-03-20T15:17:09.707 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:09.704228 I | op-osd: OSD will have its main bluestore block on "smithi045-lv-4-data-026444" 2022-03-20T15:17:10.414 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:10.411059 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:17:10.508 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:10.505359 I | op-osd: updating OSD 8 on PVC "smithi123-lv-4-data-0vfnsc" 2022-03-20T15:17:10.508 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:10.505396 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-4-data-0vfnsc" 2022-03-20T15:17:11.217 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:11.214909 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:17:11.309 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:11.306165 I | op-osd: updating OSD 9 on PVC "smithi123-lv-2-data-0l58zl" 2022-03-20T15:17:11.309 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:11.306194 I | op-osd: OSD will have its main bluestore block on "smithi123-lv-2-data-0l58zl" 2022-03-20T15:17:12.361 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:12.357717 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-20T15:17:12.361 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:17:12.357747 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-20T15:17:23.678 INFO:teuthology.orchestra.run.smithi006.stderr:+ '[' 11 '!=' 12 ']' 2022-03-20T15:17:23.678 INFO:teuthology.orchestra.run.smithi006.stdout:waiting for osd to come back up 2022-03-20T15:17:23.679 INFO:teuthology.orchestra.run.smithi006.stderr:+ echo 'waiting for osd to come back up' 2022-03-20T15:17:23.679 INFO:teuthology.orchestra.run.smithi006.stderr:++ toolbox ceph osd stat 2022-03-20T15:17:23.680 INFO:teuthology.orchestra.run.smithi006.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-20T15:17:23.680 INFO:teuthology.orchestra.run.smithi006.stderr:++ cut -f3 -d ' ' 2022-03-20T15:17:23.757 INFO:teuthology.orchestra.run.smithi006.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-20T15:17:24.320 INFO:teuthology.orchestra.run.smithi006.stderr:+ num_osd=12 2022-03-20T15:17:24.320 INFO:teuthology.orchestra.run.smithi006.stderr:+ sleep 30s 2022-03-20T15:17:54.322 INFO:teuthology.orchestra.run.smithi006.stderr:+ '[' 12 '!=' 12 ']' 2022-03-20T15:17:54.323 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-20T15:17:54.336 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch status 2022-03-20T15:17:57.991 INFO:teuthology.orchestra.run.smithi006.stdout:Backend: rook 2022-03-20T15:17:57.991 INFO:teuthology.orchestra.run.smithi006.stdout:Available: Yes 2022-03-20T15:17:58.015 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ps 2022-03-20T15:17:58.627 INFO:teuthology.orchestra.run.smithi006.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-20T15:17:58.627 INFO:teuthology.orchestra.run.smithi006.stdout:crashcollector.crash smithi006 running (4m) 0s ago 4m - - 7c0c85ada2e4 2022-03-20T15:17:58.627 INFO:teuthology.orchestra.run.smithi006.stdout:crashcollector.crash smithi045 running (5m) 0s ago 5m - - 7c0c85ada2e4 2022-03-20T15:17:58.628 INFO:teuthology.orchestra.run.smithi006.stdout:crashcollector.crash smithi123 running (5m) 0s ago 5m - - 7c0c85ada2e4 2022-03-20T15:17:58.628 INFO:teuthology.orchestra.run.smithi006.stdout:mgr.a smithi006 running (5m) 0s ago 5m - - 7c0c85ada2e4 2022-03-20T15:17:58.628 INFO:teuthology.orchestra.run.smithi006.stdout:mon.a smithi045 running (9m) 0s ago 9m - - 7c0c85ada2e4 2022-03-20T15:17:58.629 INFO:teuthology.orchestra.run.smithi006.stdout:mon.b smithi123 running (8m) 0s ago 8m - - 7c0c85ada2e4 2022-03-20T15:17:58.629 INFO:teuthology.orchestra.run.smithi006.stdout:mon.c smithi006 running (7m) 0s ago 7m - - 7c0c85ada2e4 2022-03-20T15:17:58.629 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0 smithi045 running (81s) 0s ago 81s - - 7c0c85ada2e4 2022-03-20T15:17:58.629 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1 smithi006 running (4m) 0s ago 4m - - 7c0c85ada2e4 2022-03-20T15:17:58.630 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10 smithi123 running (3m) 0s ago 3m - - 7c0c85ada2e4 2022-03-20T15:17:58.630 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11 smithi045 running (3m) 0s ago 3m - - 7c0c85ada2e4 2022-03-20T15:17:58.630 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2 smithi123 running (4m) 0s ago 4m - - 7c0c85ada2e4 2022-03-20T15:17:58.630 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3 smithi045 running (4m) 0s ago 4m - - 7c0c85ada2e4 2022-03-20T15:17:58.630 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4 smithi006 running (4m) 0s ago 4m - - 7c0c85ada2e4 2022-03-20T15:17:58.631 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5 smithi006 running (4m) 0s ago 4m - - 7c0c85ada2e4 2022-03-20T15:17:58.631 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6 smithi006 running (4m) 0s ago 4m - - 7c0c85ada2e4 2022-03-20T15:17:58.631 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7 smithi045 running (4m) 0s ago 4m - - 7c0c85ada2e4 2022-03-20T15:17:58.631 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8 smithi123 running (4m) 0s ago 4m - - 7c0c85ada2e4 2022-03-20T15:17:58.632 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9 smithi123 running (4m) 0s ago 4m - - 7c0c85ada2e4 2022-03-20T15:17:58.650 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls 2022-03-20T15:17:59.310 INFO:teuthology.orchestra.run.smithi006.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-20T15:17:59.310 INFO:teuthology.orchestra.run.smithi006.stdout:crash 3/3 0s ago 5m * 2022-03-20T15:17:59.311 INFO:teuthology.orchestra.run.smithi006.stdout:mgr 1/1 0s ago 5m count:1 2022-03-20T15:17:59.311 INFO:teuthology.orchestra.run.smithi006.stdout:mon 3/3 0s ago 9m count:3 2022-03-20T15:17:59.311 INFO:teuthology.orchestra.run.smithi006.stdout:osd 24 0s ago 4m 2022-03-20T15:17:59.311 INFO:teuthology.orchestra.run.smithi006.stdout:osd.all-available-devices 0 0s ago - * 2022-03-20T15:17:59.336 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch host ls 2022-03-20T15:17:59.837 INFO:teuthology.orchestra.run.smithi006.stdout:HOST ADDR LABELS STATUS 2022-03-20T15:17:59.838 INFO:teuthology.orchestra.run.smithi006.stdout:smithi006 172.21.15.6/smithi006 2022-03-20T15:17:59.838 INFO:teuthology.orchestra.run.smithi006.stdout:smithi045 172.21.15.45/smithi045 2022-03-20T15:17:59.838 INFO:teuthology.orchestra.run.smithi006.stdout:smithi123 172.21.15.123/smithi123 2022-03-20T15:17:59.839 INFO:teuthology.orchestra.run.smithi006.stdout:3 hosts in cluster 2022-03-20T15:17:59.860 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch device ls 2022-03-20T15:18:00.402 INFO:teuthology.orchestra.run.smithi006.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-20T15:18:00.403 INFO:teuthology.orchestra.run.smithi006.stdout:smithi006 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-20T15:18:00.403 INFO:teuthology.orchestra.run.smithi006.stdout:smithi006 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-20T15:18:00.403 INFO:teuthology.orchestra.run.smithi006.stdout:smithi006 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-20T15:18:00.403 INFO:teuthology.orchestra.run.smithi006.stdout:smithi006 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-20T15:18:00.404 INFO:teuthology.orchestra.run.smithi006.stdout:smithi045 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-20T15:18:00.404 INFO:teuthology.orchestra.run.smithi006.stdout:smithi045 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-20T15:18:00.404 INFO:teuthology.orchestra.run.smithi006.stdout:smithi045 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-20T15:18:00.404 INFO:teuthology.orchestra.run.smithi006.stdout:smithi045 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-20T15:18:00.404 INFO:teuthology.orchestra.run.smithi006.stdout:smithi123 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-20T15:18:00.405 INFO:teuthology.orchestra.run.smithi006.stdout:smithi123 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-20T15:18:00.405 INFO:teuthology.orchestra.run.smithi006.stdout:smithi123 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-20T15:18:00.405 INFO:teuthology.orchestra.run.smithi006.stdout:smithi123 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-20T15:18:00.426 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch apply rgw foo 2022-03-20T15:18:01.096 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:01.093913 I | ceph-spec: adding finalizer "cephobjectstore.ceph.rook.io" on "foo" 2022-03-20T15:18:01.097 INFO:teuthology.orchestra.run.smithi006.stdout:Created 2022-03-20T15:18:01.120 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch apply mds foo 2022-03-20T15:18:01.164 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:01.161957 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:18:01.164 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:01.162021 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e... 2022-03-20T15:18:01.772 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:01.769782 I | ceph-spec: adding finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-20T15:18:01.774 INFO:teuthology.orchestra.run.smithi006.stdout:Created 2022-03-20T15:18:01.797 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch apply rbd-mirror 2022-03-20T15:18:01.830 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:01.827979 W | ceph-file-controller: failed to set filesystem "foo" status to "". failed to update object "rook-ceph/foo" status: Operation cannot be fulfilled on cephfilesystems.ceph.rook.io "foo": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:18:01.845 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:01.843193 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:18:01.846 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:01.843273 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e... 2022-03-20T15:18:02.465 INFO:teuthology.orchestra.run.smithi006.stdout:Success 2022-03-20T15:18:02.488 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch apply nfs foo --port 12777 2022-03-20T15:18:02.521 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:02.518340 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:18:02.521 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:02.518409 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e... 2022-03-20T15:18:06.645 INFO:teuthology.orchestra.run.smithi006.stdout:Unable to create NFS daemon, check logs for more traceback 2022-03-20T15:18:06.646 INFO:teuthology.orchestra.run.smithi006.stdout:'RookOrchestrator' object has no attribute 'open_ioctx' 2022-03-20T15:18:06.667 DEBUG:teuthology.run_tasks:Unwinding manager rook 2022-03-20T15:18:06.679 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:18:07.587 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:18:07.587 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:04.000000Z", "last_refresh": "2022-03-20T15:18:07.389406Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:18:07.389406Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:18:07.389406Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:18:07.389406Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:18:07.389406Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:18:07.389406Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_name": "rbd-mirror", "service_type": "rbd-mirror", "status": {"last_refresh": "2022-03-20T15:18:07.389406Z", "running": 0, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:18:07.389406Z", "running": 0, "size": 1}}] 2022-03-20T15:18:07.606 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch rm mds.foo 2022-03-20T15:18:08.375 INFO:teuthology.orchestra.run.smithi006.stdout:Removed cephfilesystems/foo 2022-03-20T15:18:08.399 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch rm rbd-mirror 2022-03-20T15:18:09.451 INFO:teuthology.orchestra.run.smithi006.stdout:Removed cephrbdmirrors/default-rbd-mirror 2022-03-20T15:18:09.473 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch rm rgw.foo 2022-03-20T15:18:09.692 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:09.690050 I | ceph-spec: detected ceph image version: "17.0.0-11187 quincy" 2022-03-20T15:18:10.175 INFO:teuthology.orchestra.run.smithi006.stdout:Removed cephobjectstores/foo 2022-03-20T15:18:10.198 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:18:10.205 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:10.202659 W | ceph-rbd-mirror-controller: (DEPRECATED) use of peer secret names in CephRBDMirror is deprecated. Please use CephBlockPool CR to configure peer secret names and import peers. 2022-03-20T15:18:10.205 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:10.202690 I | ceph-rbd-mirror-controller: configure rbd-mirroring with 1 workers 2022-03-20T15:18:10.206 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:10.202704 I | cephclient: getting or creating ceph auth key "client.rbd-mirror.a" 2022-03-20T15:18:10.987 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:18:10.987 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:04.000000Z", "last_refresh": "2022-03-20T15:18:10.719972Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:18:10.719972Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:18:10.719972Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:18:10.719972Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:18:10.719972Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:18:10.719972Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:18:10.719972Z", "running": 0, "size": 1}}] 2022-03-20T15:18:12.022 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:12.019609 I | ceph-spec: detected ceph image version: "17.0.0-11187 quincy" 2022-03-20T15:18:12.055 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:12.051995 I | ceph-spec: detected ceph image version: "17.0.0-11187 quincy" 2022-03-20T15:18:12.060 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:12.053722 I | ceph-rbd-mirror-controller: "rook-ceph-rbd-mirror-a" deployment started 2022-03-20T15:18:12.242 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:12.239303 I | ceph-spec: object "rook-ceph-rbd-mirror-a" matched on delete, reconciling 2022-03-20T15:18:12.465 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:12.462413 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi123": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:18:13.187 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:13.184684 I | ceph-file-controller: start running mdses for filesystem "foo" 2022-03-20T15:18:13.603 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:13.601296 I | cephclient: getting or creating ceph auth key "mds.foo-a" 2022-03-20T15:18:14.079 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:14.077242 I | ceph-object-controller: reconciling object store deployments 2022-03-20T15:18:14.166 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:14.163034 I | ceph-object-controller: ceph object store gateway service running at 10.107.87.139 2022-03-20T15:18:14.166 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:14.163089 I | ceph-object-controller: reconciling object store pools 2022-03-20T15:18:14.357 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:14.354899 I | op-mds: setting mds config flags 2022-03-20T15:18:14.358 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:14.354944 I | op-config: setting "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-20T15:18:14.527 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:14.524500 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-20T15:18:15.579 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:15.577153 I | op-config: successfully set "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-20T15:18:15.590 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:15.588591 I | cephclient: getting or creating ceph auth key "mds.foo-b" 2022-03-20T15:18:15.992 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:15.989621 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi045": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:18:16.113 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:16.110853 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi045": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:18:17.324 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:17.321526 I | op-mds: setting mds config flags 2022-03-20T15:18:17.324 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:17.321545 I | op-config: setting "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-20T15:18:20.586 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:20.583921 I | op-config: successfully set "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-20T15:18:21.020 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:18:22.103 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:18:22.104 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:04.000000Z", "last_refresh": "2022-03-20T15:18:21.915866Z", "running": 5, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:18:21.915866Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:18:21.915866Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:18:21.915866Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:18:21.915866Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:18:21.915866Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:18:21.915866Z", "running": 0, "size": 1}}] 2022-03-20T15:18:22.145 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:22.142452 I | ceph-file-controller: creating filesystem "foo" 2022-03-20T15:18:23.028 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:23.025256 I | cephclient: reconciling replicated pool foo.rgw.log succeeded 2022-03-20T15:18:23.091 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:23.088427 I | cephclient: reconciling replicated pool foo.rgw.buckets.non-ec succeeded 2022-03-20T15:18:24.992 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:24.988371 I | cephclient: reconciling replicated pool foo.rgw.control succeeded 2022-03-20T15:18:24.994 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:24.990956 I | cephclient: reconciling replicated pool foo.rgw.buckets.index succeeded 2022-03-20T15:18:24.999 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:24.996701 I | cephclient: reconciling replicated pool foo.rgw.meta succeeded 2022-03-20T15:18:26.728 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:26.726188 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool ".rgw.root": failed to tag pool ".rgw.root" for application "rook-ceph-rgw": failed to get application for pool ".rgw.root": failed to get current application for pool .rgw.root: exit status 2 2022-03-20T15:18:26.758 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:26.756206 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:18:28.812 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:28.810184 I | cephclient: reconciling replicated pool foo-metadata succeeded 2022-03-20T15:18:30.276 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:30.273491 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:18:30.265+0000 7fda31058380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:18:30.277 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:18:30.265+0000 7fda31058380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:18:30.277 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:18:30.265+0000 7fda31058380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:18:30.277 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:18:30.299 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:30.296707 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:18:30.888 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:30.885537 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:18:30.877+0000 7f573c0dc380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:18:30.889 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:18:30.877+0000 7f573c0dc380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:18:30.889 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:18:30.877+0000 7f573c0dc380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:18:30.889 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:18:30.898 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:30.896466 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:18:31.460 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:31.457301 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:18:31.449+0000 7f163cc0b380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:18:31.460 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:18:31.449+0000 7f163cc0b380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:18:31.460 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:18:31.449+0000 7f163cc0b380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:18:31.461 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:18:31.540 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:31.537587 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:18:32.126 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:18:32.542 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:32.539207 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:18:32.533+0000 7fb873e06380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:18:32.542 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:18:32.533+0000 7fb873e06380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:18:32.542 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:18:32.533+0000 7fb873e06380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:18:32.543 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:18:32.577 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:32.575555 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:18:32.900 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:18:32.901 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:29.000000Z", "last_refresh": "2022-03-20T15:18:32.718316Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:18:32.718316Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:18:32.718316Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:18:32.718316Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:18:32.718316Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:18:32.718316Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:18:32.718316Z", "running": 0, "size": 1}}] 2022-03-20T15:18:33.132 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:33.129460 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:18:33.121+0000 7fa5d5cf0380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:18:33.133 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:18:33.121+0000 7fa5d5cf0380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:18:33.133 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:18:33.121+0000 7fa5d5cf0380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:18:33.134 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:18:33.139 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:33.137558 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:18:34.099 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:34.096523 I | cephclient: reconciling replicated pool foo-data0 succeeded 2022-03-20T15:18:34.126 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:34.123348 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:18:34.093+0000 7f8c59d94380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:18:34.127 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:18:34.093+0000 7f8c59d94380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:18:34.127 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:18:34.093+0000 7f8c59d94380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:18:34.127 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:18:34.152 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:34.149723 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:18:34.806 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:34.803500 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:18:34.797+0000 7fb7ac9e4380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:18:34.807 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:18:34.797+0000 7fb7ac9e4380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:18:34.807 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:18:34.797+0000 7fb7ac9e4380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:18:34.807 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:18:34.844 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:34.842161 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:18:34.914 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:34.909456 I | cephclient: creating filesystem "foo" with metadata pool "foo-metadata" and data pools [foo-data0] 2022-03-20T15:18:35.468 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:35.465060 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:18:35.457+0000 7efdecaa2380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:18:35.468 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:18:35.457+0000 7efdecaa2380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:18:35.468 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:18:35.457+0000 7efdecaa2380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:18:35.469 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:18:35.521 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:35.519076 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:18:36.170 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:36.167341 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:18:36.161+0000 7f095c91a380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:18:36.170 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:18:36.161+0000 7f095c91a380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:18:36.171 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:18:36.161+0000 7f095c91a380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:18:36.171 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:18:36.753 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:36.751232 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:18:36.870 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:36.867578 I | ceph-file-controller: created filesystem "foo" on 1 data pool(s) and metadata pool "foo-metadata" 2022-03-20T15:18:36.870 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:36.867603 I | cephclient: setting allow_standby_replay for filesystem "foo" 2022-03-20T15:18:37.326 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:37.323295 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:18:37.317+0000 7fa31381a380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:18:37.327 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:18:37.317+0000 7fa31381a380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:18:37.327 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:18:37.317+0000 7fa31381a380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:18:37.327 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:18:38.089 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:38.087031 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:18:38.094 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:38.090120 I | ceph-file-controller: CephFilesystem "rook-ceph/foo" can be deleted safely. deleting CephFilesystem "rook-ceph/foo" 2022-03-20T15:18:39.068 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:39.066066 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-20T15:18:39.601 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:39.598428 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-20T15:18:39.601 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:39.598464 I | op-mds: successfully deleted mds config for "mds.foo-a" in mon configuration database 2022-03-20T15:18:39.602 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:39.598474 I | cephclient: deleting ceph auth "mds.foo-a" 2022-03-20T15:18:40.122 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:40.119257 I | op-mds: successfully deleted mds CephX key for "mds.foo-a" 2022-03-20T15:18:40.570 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:40.567840 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-20T15:18:41.102 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:41.099729 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-20T15:18:41.103 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:41.099769 I | op-mds: successfully deleted mds config for "mds.foo-b" in mon configuration database 2022-03-20T15:18:41.103 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:41.099778 I | cephclient: deleting ceph auth "mds.foo-b" 2022-03-20T15:18:41.707 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:41.704536 I | op-mds: successfully deleted mds CephX key for "mds.foo-b" 2022-03-20T15:18:41.708 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:41.704570 I | ceph-file-controller: downing filesystem "foo" 2022-03-20T15:18:42.452 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:42.449493 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:18:42.923 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:18:43.131 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:43.129266 I | ceph-file-controller: downed filesystem "foo" 2022-03-20T15:18:43.138 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:43.135785 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:18:43.125+0000 7f96b8f74380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:18:43.138 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:18:43.125+0000 7f96b8f74380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:18:43.139 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:18:43.125+0000 7f96b8f74380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:18:43.139 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:18:43.637 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:18:43.637 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:18:43.447732Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:18:43.447732Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:18:43.447732Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:18:43.447732Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:18:43.447732Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:18:43.447732Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:18:43.447732Z", "running": 0, "size": 1}}] 2022-03-20T15:18:46.935 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:46.933119 I | cephclient: no images/snapshosts present in pool "foo-metadata" 2022-03-20T15:18:46.936 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:46.933147 I | cephclient: purging pool "foo-metadata" (id=8) 2022-03-20T15:18:49.416 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:49.413207 I | cephclient: purge completed for pool "foo-metadata" 2022-03-20T15:18:49.983 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:49.980292 I | cephclient: no images/snapshosts present in pool "foo-data0" 2022-03-20T15:18:49.983 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:49.980318 I | cephclient: purging pool "foo-data0" (id=10) 2022-03-20T15:18:51.697 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:51.695090 I | cephclient: purge completed for pool "foo-data0" 2022-03-20T15:18:51.698 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:51.695191 I | ceph-spec: removing finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-20T15:18:51.733 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:51.730971 I | ceph-spec: object "rook-ceph-mds-foo-b-keyring" matched on delete, reconciling 2022-03-20T15:18:51.809 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:51.806525 I | ceph-spec: object "rook-ceph-mds-foo-a-keyring" matched on delete, reconciling 2022-03-20T15:18:51.814 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:51.807694 I | ceph-spec: object "rook-ceph-mds-foo-b" matched on delete, reconciling 2022-03-20T15:18:51.814 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:51.808275 I | ceph-spec: object "rook-ceph-mds-foo-a" matched on delete, reconciling 2022-03-20T15:18:53.387 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:53.384545 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:18:53.663 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:18:54.004 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:54.001869 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:18:53.993+0000 7f8cf1788380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:18:54.005 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:18:53.993+0000 7f8cf1788380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:18:54.005 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:18:53.993+0000 7f8cf1788380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:18:54.005 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:18:54.367 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:18:54.368 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:18:54.188499Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:18:54.188499Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:18:54.188499Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:18:54.188499Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:18:54.188499Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:18:54.188499Z", "running": 0, "size": 1}}] 2022-03-20T15:18:55.664 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:55.662070 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi045": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:18:55.969 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:18:55.966950 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi045": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:19:04.394 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:19:05.142 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:19:05.142 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:19:04.963335Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:19:04.963335Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:19:04.963335Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:19:04.963335Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:19:04.963335Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:19:04.963335Z", "running": 0, "size": 1}}] 2022-03-20T15:19:14.492 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:19:14.490472 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:19:15.152 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:19:15.149266 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:19:15.140+0000 7ff8e9ccc380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:19:15.152 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:19:15.140+0000 7ff8e9ccc380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:19:15.153 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:19:15.140+0000 7ff8e9ccc380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:19:15.153 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:19:15.169 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:19:15.898 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:19:15.898 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:19:15.698965Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:19:15.698965Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:19:15.698965Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:19:15.698965Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:19:15.698965Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:19:15.698965Z", "running": 0, "size": 1}}] 2022-03-20T15:19:25.925 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:19:26.684 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:19:26.684 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:19:26.514047Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:19:26.514047Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:19:26.514047Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:19:26.514047Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:19:26.514047Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:19:26.514047Z", "running": 0, "size": 1}}] 2022-03-20T15:19:36.709 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:19:37.430 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:19:37.430 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:19:37.246689Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:19:37.246689Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:19:37.246689Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:19:37.246689Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:19:37.246689Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:19:37.246689Z", "running": 0, "size": 1}}] 2022-03-20T15:19:47.457 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:19:48.168 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:19:48.168 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:19:47.999090Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:19:47.999090Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:19:47.999090Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:19:47.999090Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:19:47.999090Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:19:47.999090Z", "running": 0, "size": 1}}] 2022-03-20T15:19:56.121 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:19:56.118655 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:19:56.797 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:19:56.794209 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:19:56.783+0000 7f4ad62c4380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:19:56.797 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:19:56.783+0000 7f4ad62c4380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:19:56.797 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:19:56.783+0000 7f4ad62c4380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:19:56.797 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:19:58.190 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:19:58.899 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:19:58.899 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:19:58.726531Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:19:58.726531Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:19:58.726531Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:19:58.726531Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:19:58.726531Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:19:58.726531Z", "running": 0, "size": 1}}] 2022-03-20T15:20:08.923 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:20:09.659 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:20:09.659 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:20:09.469319Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:20:09.469319Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:20:09.469319Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:20:09.469319Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:20:09.469319Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:20:09.469319Z", "running": 0, "size": 1}}] 2022-03-20T15:20:16.694 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:20:16.691611 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi123": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:20:19.685 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:20:20.475 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:20:20.475 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:20:20.246191Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:20:20.246191Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:20:20.246191Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:20:20.246191Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:20:20.246191Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:20:20.246191Z", "running": 0, "size": 1}}] 2022-03-20T15:20:30.501 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:20:31.206 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:20:31.207 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:20:31.034875Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:20:31.034875Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:20:31.034875Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:20:31.034875Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:20:31.034875Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:20:31.034875Z", "running": 0, "size": 1}}] 2022-03-20T15:20:41.232 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:20:41.958 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:20:41.959 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:20:41.779138Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:20:41.779138Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:20:41.779138Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:20:41.779138Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:20:41.779138Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:20:41.779138Z", "running": 0, "size": 1}}] 2022-03-20T15:20:51.984 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:20:53.044 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:20:53.044 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:20:52.534267Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:20:52.534267Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:20:52.534267Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:20:52.534267Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:20:52.534267Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:20:52.534267Z", "running": 0, "size": 1}}] 2022-03-20T15:21:03.075 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:21:03.834 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:21:03.834 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:21:03.615313Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:21:03.615313Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:21:03.615313Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:21:03.615313Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:21:03.615313Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:21:03.615313Z", "running": 0, "size": 1}}] 2022-03-20T15:21:13.859 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:21:14.634 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:21:14.634 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:21:14.421158Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:21:14.421158Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:21:14.421158Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:21:14.421158Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:21:14.421158Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:21:14.421158Z", "running": 0, "size": 1}}] 2022-03-20T15:21:18.725 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:21:18.723024 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:21:19.524 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:21:19.521290 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:21:19.473+0000 7f1b52303380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:21:19.524 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:21:19.473+0000 7f1b52303380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:21:19.524 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:21:19.473+0000 7f1b52303380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:21:19.524 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:21:24.658 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:21:25.366 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:21:25.367 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:21:25.194096Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:21:25.194096Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:21:25.194096Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:21:25.194096Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:21:25.194096Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:21:25.194096Z", "running": 0, "size": 1}}] 2022-03-20T15:21:35.393 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:21:36.127 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:21:36.127 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:21:35.942916Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:21:35.942916Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:21:35.942916Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:21:35.942916Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:21:35.942916Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:21:35.942916Z", "running": 0, "size": 1}}] 2022-03-20T15:21:46.154 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:21:46.900 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:21:46.900 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:21:46.733014Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:21:46.733014Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:21:46.733014Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:21:46.733014Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:21:46.733014Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:21:46.733014Z", "running": 0, "size": 1}}] 2022-03-20T15:21:56.926 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:21:57.643 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:21:57.644 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:21:57.477150Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:21:57.477150Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:21:57.477150Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:21:57.477150Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:21:57.477150Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:21:57.477150Z", "running": 0, "size": 1}}] 2022-03-20T15:22:07.666 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:22:08.345 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:22:08.345 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:22:08.172684Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:22:08.172684Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:22:08.172684Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:22:08.172684Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:22:08.172684Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:22:08.172684Z", "running": 0, "size": 1}}] 2022-03-20T15:22:18.372 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:22:19.147 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:22:19.147 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:22:18.982258Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:22:18.982258Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:22:18.982258Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:22:18.982258Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:22:18.982258Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:22:18.982258Z", "running": 0, "size": 1}}] 2022-03-20T15:22:29.178 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:22:29.924 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:22:29.925 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:22:29.756788Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:22:29.756788Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:22:29.756788Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:22:29.756788Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:22:29.756788Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:22:29.756788Z", "running": 0, "size": 1}}] 2022-03-20T15:22:39.950 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:22:40.679 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:22:40.679 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:22:40.515241Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:22:40.515241Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:22:40.515241Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:22:40.515241Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:22:40.515241Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:22:40.515241Z", "running": 0, "size": 1}}] 2022-03-20T15:22:50.706 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:22:51.448 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:22:51.449 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:22:51.250116Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:22:51.250116Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:22:51.250116Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:22:51.250116Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:22:51.250116Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:22:51.250116Z", "running": 0, "size": 1}}] 2022-03-20T15:23:01.474 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:23:02.213 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:23:02.214 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:23:02.049845Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:23:02.049845Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:23:02.049845Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:23:02.049845Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:23:02.049845Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:23:02.049845Z", "running": 0, "size": 1}}] 2022-03-20T15:23:12.242 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:23:12.970 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:23:12.970 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:23:12.782296Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:23:12.782296Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:23:12.782296Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:23:12.782296Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:23:12.782296Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:23:12.782296Z", "running": 0, "size": 1}}] 2022-03-20T15:23:22.997 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:23:23.705 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:23:23.705 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:23:23.531179Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:23:23.531179Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:23:23.531179Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:23:23.531179Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:23:23.531179Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:23:23.531179Z", "running": 0, "size": 1}}] 2022-03-20T15:23:33.730 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:23:34.549 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:23:34.549 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:23:34.287603Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:23:34.287603Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:23:34.287603Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:23:34.287603Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:23:34.287603Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:23:34.287603Z", "running": 0, "size": 1}}] 2022-03-20T15:23:44.578 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:23:45.336 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:23:45.337 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:23:45.100240Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:23:45.100240Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:23:45.100240Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:23:45.100240Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:23:45.100240Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:23:45.100240Z", "running": 0, "size": 1}}] 2022-03-20T15:23:55.363 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:23:56.052 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:23:56.053 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:23:55.890235Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:23:55.890235Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:23:55.890235Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:23:55.890235Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:23:55.890235Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:23:55.890235Z", "running": 0, "size": 1}}] 2022-03-20T15:24:03.370 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:24:03.368279 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:24:04.055 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:24:04.053103 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:24:04.042+0000 7fe7ceff8380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:24:04.056 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:24:04.042+0000 7fe7ceff8380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:24:04.056 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:24:04.042+0000 7fe7ceff8380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:24:04.056 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:24:06.080 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:24:06.812 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:24:06.813 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:24:06.641875Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:24:06.641875Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:24:06.641875Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:24:06.641875Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:24:06.641875Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:24:06.641875Z", "running": 0, "size": 1}}] 2022-03-20T15:24:16.840 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:24:17.611 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:24:17.611 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:24:17.408159Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:24:17.408159Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:24:17.408159Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:24:17.408159Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:24:17.408159Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:24:17.408159Z", "running": 0, "size": 1}}] 2022-03-20T15:24:27.642 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:24:28.361 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:24:28.362 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:24:28.170965Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:24:28.170965Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:24:28.170965Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:24:28.170965Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:24:28.170965Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:24:28.170965Z", "running": 0, "size": 1}}] 2022-03-20T15:24:38.386 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:24:39.032 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:24:39.032 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:24:38.859627Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:24:38.859627Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:24:38.859627Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:24:38.859627Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:24:38.859627Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:24:38.859627Z", "running": 0, "size": 1}}] 2022-03-20T15:24:49.057 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:24:49.850 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:24:49.851 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:24:49.586915Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:24:49.586915Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:24:49.586915Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:24:49.586915Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:24:49.586915Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:24:49.586915Z", "running": 0, "size": 1}}] 2022-03-20T15:24:59.877 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:25:00.608 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:25:00.609 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:25:00.441580Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:25:00.441580Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:25:00.441580Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:25:00.441580Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:25:00.441580Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:25:00.441580Z", "running": 0, "size": 1}}] 2022-03-20T15:25:10.632 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:25:11.349 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:25:11.349 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:25:11.176214Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:25:11.176214Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:25:11.176214Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:25:11.176214Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:25:11.176214Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:25:11.176214Z", "running": 0, "size": 1}}] 2022-03-20T15:25:21.375 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:25:22.078 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:25:22.078 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:25:21.898768Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:25:21.898768Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:25:21.898768Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:25:21.898768Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:25:21.898768Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:25:21.898768Z", "running": 0, "size": 1}}] 2022-03-20T15:25:32.111 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:25:32.819 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:25:32.820 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:25:32.665474Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:25:32.665474Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:25:32.665474Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:25:32.665474Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:25:32.665474Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:25:32.665474Z", "running": 0, "size": 1}}] 2022-03-20T15:25:42.846 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:25:43.915 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:25:43.915 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:25:43.454848Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:25:43.454848Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:25:43.454848Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:25:43.454848Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:25:43.454848Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:25:43.454848Z", "running": 0, "size": 1}}] 2022-03-20T15:25:53.941 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:25:54.655 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:25:54.656 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:25:54.472895Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:25:54.472895Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:25:54.472895Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:25:54.472895Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:25:54.472895Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:25:54.472895Z", "running": 0, "size": 1}}] 2022-03-20T15:26:04.685 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:26:05.476 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:26:05.477 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:26:05.277619Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:26:05.277619Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:26:05.277619Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:26:05.277619Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:26:05.277619Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:26:05.277619Z", "running": 0, "size": 1}}] 2022-03-20T15:26:15.502 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:26:16.196 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:26:16.197 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:26:16.031945Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:26:16.031945Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:26:16.031945Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:26:16.031945Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:26:16.031945Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:26:16.031945Z", "running": 0, "size": 1}}] 2022-03-20T15:26:26.225 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:26:26.972 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:26:26.972 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:26:26.784701Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:26:26.784701Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:26:26.784701Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:26:26.784701Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:26:26.784701Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:26:26.784701Z", "running": 0, "size": 1}}] 2022-03-20T15:26:36.997 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:26:37.764 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:26:37.764 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:26:37.580421Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:26:37.580421Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:26:37.580421Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:26:37.580421Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:26:37.580421Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:26:37.580421Z", "running": 0, "size": 1}}] 2022-03-20T15:26:47.789 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:26:48.469 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:26:48.470 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:26:48.297943Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:26:48.297943Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:26:48.297943Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:26:48.297943Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:26:48.297943Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:26:48.297943Z", "running": 0, "size": 1}}] 2022-03-20T15:26:58.509 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:26:59.250 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:26:59.250 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:26:59.042409Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:26:59.042409Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:26:59.042409Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:26:59.042409Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:26:59.042409Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:26:59.042409Z", "running": 0, "size": 1}}] 2022-03-20T15:27:09.273 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:27:09.987 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:27:09.987 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:27:09.791671Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:27:09.791671Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:27:09.791671Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:27:09.791671Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:27:09.791671Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:27:09.791671Z", "running": 0, "size": 1}}] 2022-03-20T15:27:20.012 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:27:20.750 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:27:20.751 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:27:20.577259Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:27:20.577259Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:27:20.577259Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:27:20.577259Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:27:20.577259Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:27:20.577259Z", "running": 0, "size": 1}}] 2022-03-20T15:27:30.777 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:27:31.550 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:27:31.550 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:27:31.339011Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:27:31.339011Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:27:31.339011Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:27:31.339011Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:27:31.339011Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:27:31.339011Z", "running": 0, "size": 1}}] 2022-03-20T15:27:41.575 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:27:42.308 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:27:42.309 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:27:42.120302Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:27:42.120302Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:27:42.120302Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:27:42.120302Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:27:42.120302Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:27:42.120302Z", "running": 0, "size": 1}}] 2022-03-20T15:27:52.334 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:27:53.074 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:27:53.074 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:27:52.873125Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:27:52.873125Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:27:52.873125Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:27:52.873125Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:27:52.873125Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:27:52.873125Z", "running": 0, "size": 1}}] 2022-03-20T15:28:03.113 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:28:06.991 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:28:06.991 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:28:06.822001Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:28:06.822001Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:28:06.822001Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:28:06.822001Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:28:06.822001Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:28:06.822001Z", "running": 0, "size": 1}}] 2022-03-20T15:28:17.018 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:28:17.732 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:28:17.732 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:28:17.542329Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:28:17.542329Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:28:17.542329Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:28:17.542329Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:28:17.542329Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:28:17.542329Z", "running": 0, "size": 1}}] 2022-03-20T15:28:27.753 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:28:28.500 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:28:28.500 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:28:28.294715Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:28:28.294715Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:28:28.294715Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:28:28.294715Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:28:28.294715Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:28:28.294715Z", "running": 0, "size": 1}}] 2022-03-20T15:28:38.525 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:28:39.240 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:28:39.241 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:28:39.061472Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:28:39.061472Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:28:39.061472Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:28:39.061472Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:28:39.061472Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:28:39.061472Z", "running": 0, "size": 1}}] 2022-03-20T15:28:49.266 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:28:50.014 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:28:50.014 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:28:49.833911Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:28:49.833911Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:28:49.833911Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:28:49.833911Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:28:49.833911Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:28:49.833911Z", "running": 0, "size": 1}}] 2022-03-20T15:29:00.041 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:29:00.777 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:29:00.777 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:29:00.597564Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:29:00.597564Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:29:00.597564Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:29:00.597564Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:29:00.597564Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:29:00.597564Z", "running": 0, "size": 1}}] 2022-03-20T15:29:10.811 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:29:11.526 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:29:11.526 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:29:11.349361Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:29:11.349361Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:29:11.349361Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:29:11.349361Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:29:11.349361Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:29:11.349361Z", "running": 0, "size": 1}}] 2022-03-20T15:29:21.555 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:29:22.344 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:29:22.344 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:29:22.069269Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:29:22.069269Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:29:22.069269Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:29:22.069269Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:29:22.069269Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:29:22.069269Z", "running": 0, "size": 1}}] 2022-03-20T15:29:31.742 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:29:31.740886 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:29:32.396 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:29:32.468 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:29:32.466436 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:29:32.455+0000 7f938ce00380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:29:32.468 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:29:32.455+0000 7f938ce00380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:29:32.468 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:29:32.455+0000 7f938ce00380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:29:32.469 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:29:33.183 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:29:33.183 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:29:33.001869Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:29:33.001869Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:29:33.001869Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:29:33.001869Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:29:33.001869Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:29:33.001869Z", "running": 0, "size": 1}}] 2022-03-20T15:29:43.210 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:29:43.938 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:29:43.939 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:29:43.735213Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:29:43.735213Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:29:43.735213Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:29:43.735213Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:29:43.735213Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:29:43.735213Z", "running": 0, "size": 1}}] 2022-03-20T15:29:53.969 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:29:54.746 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:29:54.747 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:29:54.495404Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:29:54.495404Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:29:54.495404Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:29:54.495404Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:29:54.495404Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:29:54.495404Z", "running": 0, "size": 1}}] 2022-03-20T15:30:04.770 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:30:05.499 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:30:05.499 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:30:05.321275Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:30:05.321275Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:30:05.321275Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:30:05.321275Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:30:05.321275Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:30:05.321275Z", "running": 0, "size": 1}}] 2022-03-20T15:30:15.524 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:30:16.288 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:30:16.288 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:30:16.117932Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:30:16.117932Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:30:16.117932Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:30:16.117932Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:30:16.117932Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:30:16.117932Z", "running": 0, "size": 1}}] 2022-03-20T15:30:26.315 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:30:27.066 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:30:27.066 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:30:26.889225Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:30:26.889225Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:30:26.889225Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:30:26.889225Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:30:26.889225Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:30:26.889225Z", "running": 0, "size": 1}}] 2022-03-20T15:30:37.092 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:30:37.839 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:30:37.839 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:30:37.653898Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:30:37.653898Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:30:37.653898Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:30:37.653898Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:30:37.653898Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:30:37.653898Z", "running": 0, "size": 1}}] 2022-03-20T15:30:47.867 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:30:48.592 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:30:48.592 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:30:48.411115Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:30:48.411115Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:30:48.411115Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:30:48.411115Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:30:48.411115Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:30:48.411115Z", "running": 0, "size": 1}}] 2022-03-20T15:30:58.614 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:30:59.348 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:30:59.348 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:30:59.164000Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:30:59.164000Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:30:59.164000Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:30:59.164000Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:30:59.164000Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:30:59.164000Z", "running": 0, "size": 1}}] 2022-03-20T15:31:09.375 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:31:10.066 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:31:10.066 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:31:09.862545Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:31:09.862545Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:31:09.862545Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:31:09.862545Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:31:09.862545Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:31:09.862545Z", "running": 0, "size": 1}}] 2022-03-20T15:31:20.091 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:31:20.784 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:31:20.784 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:31:20.612704Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:31:20.612704Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:31:20.612704Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:31:20.612704Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:31:20.612704Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:31:20.612704Z", "running": 0, "size": 1}}] 2022-03-20T15:31:30.809 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:31:31.614 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:31:31.615 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:31:31.464866Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:31:31.464866Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:31:31.464866Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:31:31.464866Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:31:31.464866Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:31:31.464866Z", "running": 0, "size": 1}}] 2022-03-20T15:31:41.635 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:31:42.344 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:31:42.345 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:31:42.165487Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:31:42.165487Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:31:42.165487Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:31:42.165487Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:31:42.165487Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:31:42.165487Z", "running": 0, "size": 1}}] 2022-03-20T15:31:52.389 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:31:53.105 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:31:53.106 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:31:52.912402Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:31:52.912402Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:31:52.912402Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:31:52.912402Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:31:52.912402Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:31:52.912402Z", "running": 0, "size": 1}}] 2022-03-20T15:32:03.130 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:32:03.999 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:32:04.000 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:32:03.824695Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:32:03.824695Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:32:03.824695Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:32:03.824695Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:32:03.824695Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:32:03.824695Z", "running": 0, "size": 1}}] 2022-03-20T15:32:14.025 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:32:14.850 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:32:14.850 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:32:14.571250Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:32:14.571250Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:32:14.571250Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:32:14.571250Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:32:14.571250Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:32:14.571250Z", "running": 0, "size": 1}}] 2022-03-20T15:32:24.870 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:32:25.633 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:32:25.633 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:32:25.456022Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:32:25.456022Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:32:25.456022Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:32:25.456022Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:32:25.456022Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:32:25.456022Z", "running": 0, "size": 1}}] 2022-03-20T15:32:35.656 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:32:36.421 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:32:36.422 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:32:36.241170Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:32:36.241170Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:32:36.241170Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:32:36.241170Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:32:36.241170Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:32:36.241170Z", "running": 0, "size": 1}}] 2022-03-20T15:32:46.448 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:32:47.142 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:32:47.142 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:32:46.967445Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:32:46.967445Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:32:46.967445Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:32:46.967445Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:32:46.967445Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:32:46.967445Z", "running": 0, "size": 1}}] 2022-03-20T15:32:57.166 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:32:57.950 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:32:57.950 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:32:57.760078Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:32:57.760078Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:32:57.760078Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:32:57.760078Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:32:57.760078Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:32:57.760078Z", "running": 0, "size": 1}}] 2022-03-20T15:33:07.975 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:33:08.865 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:33:08.865 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:33:08.594102Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:33:08.594102Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:33:08.594102Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:33:08.594102Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:33:08.594102Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:33:08.594102Z", "running": 0, "size": 1}}] 2022-03-20T15:33:18.889 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:33:19.622 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:33:19.623 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:33:19.446587Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:33:19.446587Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:33:19.446587Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:33:19.446587Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:33:19.446587Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:33:19.446587Z", "running": 0, "size": 1}}] 2022-03-20T15:33:29.646 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:33:30.435 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:33:30.435 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:33:30.161284Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:33:30.161284Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:33:30.161284Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:33:30.161284Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:33:30.161284Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:33:30.161284Z", "running": 0, "size": 1}}] 2022-03-20T15:33:40.462 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:33:41.180 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:33:41.181 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:33:41.019977Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:33:41.019977Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:33:41.019977Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:33:41.019977Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:33:41.019977Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:33:41.019977Z", "running": 0, "size": 1}}] 2022-03-20T15:33:51.206 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:33:51.896 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:33:51.897 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:33:51.725996Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:33:51.725996Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:33:51.725996Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:33:51.725996Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:33:51.725996Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:33:51.725996Z", "running": 0, "size": 1}}] 2022-03-20T15:34:01.923 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:34:02.693 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:34:02.694 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:34:02.494939Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:34:02.494939Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:34:02.494939Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:34:02.494939Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:34:02.494939Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:34:02.494939Z", "running": 0, "size": 1}}] 2022-03-20T15:34:12.721 DEBUG:teuthology.orchestra.run.smithi006:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-54sh5 -- ceph orch ls -f json 2022-03-20T15:34:13.413 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-20T15:34:13.413 INFO:teuthology.orchestra.run.smithi006.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:36.000000Z", "last_refresh": "2022-03-20T15:34:13.256046Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:12:00.000000Z", "last_refresh": "2022-03-20T15:34:13.256046Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:08:58.000000Z", "last_refresh": "2022-03-20T15:34:13.256046Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T15:13:35.000000Z", "last_refresh": "2022-03-20T15:34:13.256046Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:34:13.256046Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:34:13.256046Z", "running": 0, "size": 1}}] 2022-03-20T15:34:13.438 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-20T15:34:13.439 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks/rook.py", line 530, in ceph_config_keyring yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-20T15:34:13.439 INFO:tasks.rook:Cleaning up config and client.admin keyring 2022-03-20T15:34:13.440 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-20T15:34:13.455 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-20T15:34:13.471 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-20T15:34:13.488 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks/rook.py", line 478, in rook_post_config yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-20T15:34:13.489 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks/rook.py", line 442, in rook_toolbox yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-20T15:34:13.497 DEBUG:teuthology.orchestra.remote:smithi006:rook/deploy/examples/operator.yaml is 20KB 2022-03-20T15:34:13.550 DEBUG:teuthology.orchestra.run.smithi006:> kubectl delete -f rook/deploy/examples/toolbox.yaml 2022-03-20T15:34:13.614 INFO:teuthology.orchestra.run.smithi006.stdout:deployment.apps "rook-ceph-tools" deleted 2022-03-20T15:34:13.628 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks/rook.py", line 379, in rook_cluster yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-20T15:34:13.629 DEBUG:teuthology.orchestra.run.smithi006:> kubectl delete -f cluster.yaml 2022-03-20T15:34:13.745 INFO:teuthology.orchestra.run.smithi006.stdout:cephcluster.ceph.rook.io "rook-ceph" deleted 2022-03-20T15:34:13.751 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:13.751419 I | ceph-cluster-controller: CR "rook-ceph" is going be deleted, cancelling any ongoing orchestration 2022-03-20T15:34:13.751 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:13.751540 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-20T15:34:13.751 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:13.751935 I | operator: successfully started the controller-runtime manager 2022-03-20T15:34:13.756 INFO:tasks.rook.operator.smithi006.stdout:I0320 15:34:13.752083 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-20T15:34:13.768 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:13.769552 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-20T15:34:13.769 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:13.769570 I | operator: watching all namespaces for Ceph CRs 2022-03-20T15:34:13.769 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:13.769608 I | operator: setting up schemes 2022-03-20T15:34:13.772 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:13.773215 I | operator: setting up the controller-runtime manager 2022-03-20T15:34:14.076 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:14.076754 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-20T15:34:14.080 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:14.079660 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-20T15:34:14.081 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:14.079696 I | ceph-cluster-controller: successfully started 2022-03-20T15:34:14.081 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:14.079751 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-20T15:34:14.081 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:14.079776 I | ceph-crashcollector-controller: successfully started 2022-03-20T15:34:14.081 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:14.079802 I | ceph-block-pool-controller: successfully started 2022-03-20T15:34:14.082 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:14.079841 I | ceph-object-store-user-controller: successfully started 2022-03-20T15:34:14.082 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:14.079877 I | ceph-object-realm-controller: successfully started 2022-03-20T15:34:14.084 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:14.079896 I | ceph-object-zonegroup-controller: successfully started 2022-03-20T15:34:14.084 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:14.079907 I | ceph-object-zone-controller: successfully started 2022-03-20T15:34:14.084 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:14.080043 I | ceph-object-controller: successfully started 2022-03-20T15:34:14.085 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:14.080081 I | ceph-file-controller: successfully started 2022-03-20T15:34:14.085 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:14.080118 I | ceph-nfs-controller: successfully started 2022-03-20T15:34:14.085 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:14.080146 I | ceph-rbd-mirror-controller: successfully started 2022-03-20T15:34:14.085 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:14.080173 I | ceph-client-controller: successfully started 2022-03-20T15:34:14.085 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:14.080194 I | ceph-filesystem-mirror-controller: successfully started 2022-03-20T15:34:14.086 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:14.080220 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-20T15:34:14.086 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:14.080239 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-20T15:34:14.086 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:14.080257 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-20T15:34:14.087 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:14.080275 I | ceph-bucket-topic: successfully started 2022-03-20T15:34:14.087 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:14.080292 I | ceph-bucket-notification: successfully started 2022-03-20T15:34:14.087 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:14.080304 I | ceph-bucket-notification: successfully started 2022-03-20T15:34:14.087 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:14.080315 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-20T15:34:14.087 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:14.081549 I | operator: starting the controller-runtime manager 2022-03-20T15:34:14.189 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:14.186810 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-20T15:34:14.190 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:14.186880 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-20T15:34:14.190 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:14.186901 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-20T15:34:14.194 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:14.190997 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:34:14.194 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:14.191574 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-20T15:34:14.287 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:14.288630 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:34:14.300 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:14.301030 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:34:14.867 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:14.867108 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:34:14.854+0000 7f2a42c6f380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:34:14.867 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:34:14.854+0000 7f2a42c6f380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:34:14.867 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:34:14.854+0000 7f2a42c6f380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:34:14.868 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:34:14.877 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:14.878195 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:34:15.571 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:15.570762 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:34:15.558+0000 7f45a9ab6380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:34:15.571 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:34:15.558+0000 7f45a9ab6380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:34:15.571 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:34:15.558+0000 7f45a9ab6380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:34:15.571 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:34:15.607 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:15.608188 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:34:16.256 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:16.256531 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:34:16.246+0000 7fed47f61380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:34:16.256 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:34:16.246+0000 7fed47f61380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:34:16.257 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:34:16.246+0000 7fed47f61380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:34:16.257 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:34:16.282 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:16.282993 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:34:16.975 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:16.975558 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:34:16.962+0000 7f1a91513380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:34:16.976 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:34:16.962+0000 7f1a91513380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:34:16.976 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:34:16.962+0000 7f1a91513380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:34:16.976 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:34:17.023 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:17.024028 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:34:17.679 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:17.679532 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:34:17.666+0000 7f6bb488a380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:34:17.680 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:34:17.666+0000 7f6bb488a380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:34:17.680 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:34:17.666+0000 7f6bb488a380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:34:17.680 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:34:17.811 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:17.811620 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:34:18.508 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:18.509186 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:34:18.454+0000 7fb23d6f1380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:34:18.509 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:34:18.454+0000 7fb23d6f1380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:34:18.509 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:34:18.454+0000 7fb23d6f1380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:34:18.509 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:34:18.676 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:18.677226 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:34:19.385 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:19.385143 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:34:19.374+0000 7fccb73be380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:34:19.385 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:34:19.374+0000 7fccb73be380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:34:19.385 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:34:19.374+0000 7fccb73be380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:34:19.386 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:34:19.711 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:19.712420 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:34:20.449 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:20.450236 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:34:20.438+0000 7f6afa251380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:34:20.450 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:34:20.438+0000 7f6afa251380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:34:20.450 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:34:20.438+0000 7f6afa251380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:34:20.450 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:34:21.096 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:21.097410 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:34:21.762 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:21.763124 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:34:21.750+0000 7f500adca380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:34:21.763 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:34:21.750+0000 7f500adca380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:34:21.763 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:34:21.750+0000 7f500adca380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:34:21.763 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:34:23.052 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:23.052705 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:34:23.772 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:23.772502 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:34:23.762+0000 7f23079db380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:34:23.772 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:34:23.762+0000 7f23079db380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:34:23.772 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:34:23.762+0000 7f23079db380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:34:23.772 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:34:24.413 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:24.413698 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:34:24.425 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:24.426619 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:34:26.340 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:26.341200 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:34:27.007 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:27.007341 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:34:26.994+0000 7f43e8abe380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:34:27.007 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:34:26.994+0000 7f43e8abe380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:34:27.008 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:34:26.994+0000 7f43e8abe380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:34:27.008 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:34:32.136 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:32.136986 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:34:32.805 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:32.805707 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:34:32.794+0000 7f050fa85380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:34:32.806 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:34:32.794+0000 7f050fa85380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:34:32.806 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:34:32.794+0000 7f050fa85380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:34:32.806 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:34:34.603 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:34.604290 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:34:34.617 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:34.618090 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:34:43.055 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:43.055683 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:34:43.741 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:43.741745 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:34:43.730+0000 7f996cbb0380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:34:43.742 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:34:43.730+0000 7f996cbb0380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:34:43.742 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:34:43.730+0000 7f996cbb0380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:34:43.742 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:34:44.782 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:44.783610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:34:44.794 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:44.795167 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:34:54.893 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:54.894303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:34:54.905 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:34:54.906529 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:35:04.229 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:35:04.230465 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:35:04.922 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:35:04.922983 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:35:04.913+0000 7f35a065d380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:35:04.922 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:35:04.913+0000 7f35a065d380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:35:04.923 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:35:04.913+0000 7f35a065d380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:35:04.923 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:35:04.981 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:35:04.982456 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:35:04.996 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:35:04.997281 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:35:15.104 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:35:15.105495 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:35:15.116 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:35:15.117763 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:35:25.260 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:35:25.261269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:35:25.270 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:35:25.271427 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:35:35.338 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:35:35.339292 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:35:35.430 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:35:35.430843 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:35:45.540 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:35:45.541676 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:35:45.554 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:35:45.555691 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:35:45.888 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:35:45.889597 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:35:46.560 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:35:46.561502 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:35:46.548+0000 7f526b997380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:35:46.561 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:35:46.548+0000 7f526b997380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:35:46.561 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:35:46.548+0000 7f526b997380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:35:46.561 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:35:55.622 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:35:55.623856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:35:55.635 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:35:55.637111 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:36:05.761 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:36:05.762489 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:36:05.774 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:36:05.775854 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:36:15.900 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:36:15.901099 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:36:15.912 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:36:15.914126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:36:26.023 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:36:26.024481 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:36:26.037 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:36:26.038669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:36:36.104 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:36:36.104847 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:36:36.117 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:36:36.118400 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:36:46.228 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:36:46.229002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:36:46.240 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:36:46.241965 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:36:56.306 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:36:56.306985 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:36:56.372 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:36:56.372949 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:37:06.516 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:37:06.517316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:37:06.529 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:37:06.530561 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:37:08.509 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:37:08.510228 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:37:09.166 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:37:09.167363 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:37:09.155+0000 7f44e5150380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:37:09.167 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:37:09.155+0000 7f44e5150380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:37:09.167 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:37:09.155+0000 7f44e5150380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:37:09.167 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:37:16.630 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:37:16.631045 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:37:16.641 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:37:16.642429 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:37:26.713 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:37:26.714203 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:37:26.783 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:37:26.784420 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:37:36.864 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:37:36.865539 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:37:36.926 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:37:36.927877 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:37:47.038 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:37:47.039260 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:37:47.051 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:37:47.052634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:37:57.117 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:37:57.114113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:37:57.125 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:37:57.126404 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:38:07.262 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:38:07.263267 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:38:07.276 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:38:07.277590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:38:17.366 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:38:17.367199 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:38:17.399 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:38:17.400021 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:38:27.465 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:38:27.467040 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:38:27.483 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:38:27.484904 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:38:37.596 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:38:37.596972 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:38:37.609 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:38:37.610982 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:38:47.680 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:38:47.681690 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:38:47.690 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:38:47.691972 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:38:57.847 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:38:57.848157 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:38:57.860 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:38:57.861841 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:39:07.967 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:39:07.968927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:39:07.977 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:39:07.979348 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:39:18.040 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:39:18.038915 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:39:18.050 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:39:18.051769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:39:28.148 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:39:28.149046 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:39:28.159 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:39:28.161024 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:39:38.243 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:39:38.244849 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:39:38.257 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:39:38.258535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:39:48.360 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:39:48.362158 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:39:48.375 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:39:48.376798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:39:53.014 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:39:53.015815 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:39:53.673 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:39:53.674525 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:39:53.663+0000 7feb58b11380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:39:53.674 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:39:53.663+0000 7feb58b11380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:39:53.674 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:39:53.663+0000 7feb58b11380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:39:53.674 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:39:58.501 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:39:58.502549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:39:58.514 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:39:58.515981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:40:08.607 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:40:08.607697 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:40:08.621 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:40:08.623117 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:40:18.679 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:40:18.680183 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:40:18.693 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:40:18.695042 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:40:28.832 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:40:28.833134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:40:28.843 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:40:28.844953 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:40:38.947 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:40:38.948801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:40:38.960 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:40:38.961608 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:40:49.057 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:40:49.058163 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:40:49.070 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:40:49.071880 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:40:59.226 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:40:59.227746 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:40:59.240 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:40:59.242040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:41:09.354 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:41:09.355027 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:41:09.368 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:41:09.369509 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:41:19.437 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:41:19.438036 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:41:19.449 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:41:19.450322 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:41:29.548 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:41:29.548956 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:41:29.562 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:41:29.563476 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:41:39.665 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:41:39.666421 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:41:39.679 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:41:39.680009 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:41:49.752 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:41:49.752730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:41:49.766 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:41:49.767223 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:41:59.882 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:41:59.883146 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:41:59.897 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:41:59.898440 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:42:09.992 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:42:09.993376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:42:10.005 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:42:10.006148 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:42:20.084 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:42:20.085131 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:42:20.096 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:42:20.097646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:42:30.202 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:42:30.202824 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:42:30.215 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:42:30.215941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:42:40.277 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:42:40.278591 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:42:40.298 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:42:40.299223 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:42:50.443 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:42:50.443834 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:42:50.453 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:42:50.454103 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:43:00.584 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:43:00.584732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:43:00.601 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:43:00.602472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:43:10.687 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:43:10.687965 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:43:10.700 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:43:10.701861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:43:20.779 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:43:20.780208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:43:20.830 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:43:20.830603 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:43:30.905 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:43:30.906188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:43:30.965 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:43:30.966166 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:43:41.047 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:43:41.047926 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:43:41.061 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:43:41.062108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:43:51.175 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:43:51.175761 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:43:51.187 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:43:51.188196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:44:01.268 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:44:01.268716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:44:01.282 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:44:01.283434 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:44:11.418 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:44:11.419110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:44:11.432 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:44:11.433523 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:44:21.503 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:44:21.504590 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:44:21.515 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:44:21.516337 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:44:31.618 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:44:31.619321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:44:31.630 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:44:31.631472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:44:41.730 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:44:41.730805 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:44:41.743 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:44:41.744013 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:44:51.873 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:44:51.874144 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:44:51.887 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:44:51.888571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:45:01.996 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:45:01.997705 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:45:02.009 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:45:02.010519 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:45:12.108 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:45:12.109160 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:45:12.122 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:45:12.123584 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:45:21.365 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:45:21.365876 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:45:22.069 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:45:22.069682 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:45:22.059+0000 7f3fbad21380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:45:22.070 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:45:22.059+0000 7f3fbad21380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:45:22.070 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:45:22.059+0000 7f3fbad21380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:45:22.070 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:45:22.197 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:45:22.197976 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:45:22.227 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:45:22.227570 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:45:32.321 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:45:32.322017 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:45:32.335 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:45:32.335701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:45:42.468 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:45:42.469024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:45:42.480 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:45:42.481193 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:45:52.549 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:45:52.550365 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:45:52.561 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:45:52.562432 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:46:02.670 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:46:02.670743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:46:02.686 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:46:02.685300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:46:12.787 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:46:12.787495 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:46:12.798 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:46:12.798963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:46:22.904 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:46:22.904680 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:46:22.918 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:46:22.919027 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:46:33.023 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:46:33.023560 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:46:33.037 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:46:33.038061 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:46:43.154 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:46:43.154613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:46:43.166 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:46:43.167443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:46:53.236 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:46:53.236480 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:46:53.306 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:46:53.307029 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:47:03.428 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:47:03.428988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:47:03.441 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:47:03.442477 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:47:13.535 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:47:13.535040 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:47:13.547 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:47:13.547683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:47:23.650 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:47:23.649845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:47:23.664 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:47:23.663996 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:47:33.758 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:47:33.758233 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:47:33.768 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:47:33.769024 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:47:43.880 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:47:43.880769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:47:43.893 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:47:43.893768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:47:53.978 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:47:53.979077 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:47:53.992 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:47:53.993087 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:48:04.058 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:48:04.059072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:48:04.072 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:48:04.072549 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:48:14.167 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:48:14.168107 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:48:14.181 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:48:14.181790 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:48:24.258 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:48:24.258780 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:48:24.273 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:48:24.273997 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:48:34.341 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:48:34.341891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:48:34.406 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:48:34.406507 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:48:44.505 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:48:44.505070 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:48:44.519 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:48:44.519810 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:48:54.586 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:48:54.584670 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:48:54.595 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:48:54.596024 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:49:04.703 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:49:04.703079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:49:04.717 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:49:04.717910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:49:14.783 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:49:14.783672 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:49:14.797 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:49:14.797812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:49:24.873 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:49:24.873231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:49:24.885 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:49:24.885281 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:49:34.984 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:49:34.984488 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:49:34.998 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:49:34.999363 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:49:45.072 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:49:45.072366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:49:45.081 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:49:45.081952 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:49:55.155 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:49:55.155546 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:49:55.168 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:49:55.168600 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:50:05.262 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:50:05.262637 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:50:05.278 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:50:05.278827 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:50:15.380 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:50:15.380080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:50:15.394 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:50:15.394364 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:50:25.534 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:50:25.534577 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:50:25.548 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:50:25.548864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:50:35.697 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:50:35.697753 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:50:35.712 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:50:35.712995 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:50:45.794 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:50:45.794185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:50:45.851 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:50:45.851609 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:50:56.026 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:50:56.026295 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:50:56.039 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:50:56.039791 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:51:06.134 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:51:06.133494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:51:06.148 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:51:06.149185 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:51:16.251 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:51:16.250927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:51:16.262 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:51:16.263114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:51:26.487 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:51:26.487335 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:51:26.499 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:51:26.499992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:51:36.593 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:51:36.593722 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:51:36.603 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:51:36.603763 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:51:46.746 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:51:46.746566 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:51:46.758 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:51:46.759328 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:51:56.857 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:51:56.857267 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:51:56.869 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:51:56.869987 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:52:06.954 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:52:06.954499 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:52:06.969 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:52:06.969438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:52:17.090 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:52:17.090460 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:52:17.100 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:52:17.101301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:52:27.232 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:52:27.232765 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:52:27.244 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:52:27.245274 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:52:37.360 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:52:37.360791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:52:37.373 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:52:37.373808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:52:47.431 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:52:47.432094 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:52:47.445 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:52:47.445260 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:52:57.541 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:52:57.541169 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:52:57.554 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:52:57.555123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:53:07.635 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:53:07.636052 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:53:07.695 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:53:07.695900 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:53:17.782 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:53:17.782518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:53:17.846 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:53:17.842720 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:53:27.942 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:53:27.942751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:53:27.958 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:53:27.959274 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:53:38.018 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:53:38.018952 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:53:38.033 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:53:38.033701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:53:48.136 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:53:48.137008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:53:48.170 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:53:48.171119 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:53:58.238 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:53:58.237081 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:53:58.250 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:53:58.250442 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:54:08.326 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:54:08.326999 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:54:08.337 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:54:08.337913 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:54:18.440 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:54:18.440531 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:54:18.452 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:54:18.453220 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:54:28.587 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:54:28.586780 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:54:28.608 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:54:28.608843 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:54:38.691 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:54:38.691843 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:54:38.704 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:54:38.705223 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:54:48.771 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:54:48.771912 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:54:48.782 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:54:48.783177 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:54:58.859 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:54:58.859614 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:54:58.874 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:54:58.874491 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:55:08.938 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:55:08.938548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:55:08.956 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:55:08.953084 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:55:19.030 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:55:19.030587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:55:19.043 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:55:19.044148 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:55:29.171 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:55:29.172305 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:55:29.185 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:55:29.186419 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:55:39.241 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:55:39.242166 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:55:39.250 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:55:39.251315 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:55:49.315 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:55:49.315400 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:55:49.402 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:55:49.402869 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:55:59.509 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:55:59.509443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:55:59.522 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:55:59.522764 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:56:09.633 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:56:09.633481 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:56:09.645 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:56:09.645884 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:56:17.437 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:56:17.437883 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T15:56:18.152 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:56:18.152721 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:56:18.139+0000 7f474f753380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:56:18.153 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:56:18.143+0000 7f474f753380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:56:18.153 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T15:56:18.143+0000 7f474f753380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:56:18.153 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:56:19.722 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:56:19.723319 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:56:19.782 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:56:19.783539 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:56:29.859 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:56:29.859736 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:56:29.877 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:56:29.878210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:56:39.936 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:56:39.937047 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:56:39.949 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:56:39.949536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:56:50.072 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:56:50.073508 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:56:50.087 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:56:50.087793 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:57:00.159 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:57:00.159776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:57:00.170 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:57:00.171797 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:57:10.304 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:57:10.304580 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:57:10.319 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:57:10.318079 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:57:20.438 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:57:20.438960 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:57:20.468 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:57:20.469056 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:57:30.533 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:57:30.534364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:57:30.545 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:57:30.546808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:57:40.649 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:57:40.650014 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:57:40.660 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:57:40.661578 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:57:50.768 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:57:50.769100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:57:50.780 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:57:50.781327 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:58:00.851 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:58:00.851962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:58:00.908 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:58:00.909723 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:58:11.028 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:58:11.029189 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:58:11.043 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:58:11.043999 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:58:21.114 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:58:21.114334 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:58:21.125 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:58:21.126959 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:58:31.193 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:58:31.194644 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:58:31.243 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:58:31.243930 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:58:41.354 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:58:41.354698 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:58:41.384 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:58:41.385676 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:58:51.497 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:58:51.497795 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:58:51.512 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:58:51.511589 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:59:01.587 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:59:01.588338 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:59:01.600 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:59:01.601590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:59:11.695 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:59:11.696129 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:59:11.709 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:59:11.710880 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:59:21.824 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:59:21.824748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:59:21.838 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:59:21.839986 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:59:31.912 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:59:31.913441 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:59:31.923 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:59:31.924536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:59:42.272 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:59:42.272925 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:59:42.283 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:59:42.284963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:59:52.391 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:59:52.391938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:59:52.405 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 15:59:52.406671 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:00:02.503 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:00:02.503899 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:00:02.517 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:00:02.518954 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:00:12.589 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:00:12.590657 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:00:12.603 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:00:12.604717 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:00:22.723 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:00:22.724317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:00:22.740 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:00:22.741156 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:00:32.882 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:00:32.883068 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:00:32.895 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:00:32.896168 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:00:42.952 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:00:42.953995 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:00:42.965 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:00:42.966467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:00:53.094 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:00:53.095461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:00:53.110 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:00:53.111464 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:01:03.204 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:01:03.204600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:01:03.215 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:01:03.216963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:01:13.317 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:01:13.317522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:01:13.329 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:01:13.331210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:01:23.443 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:01:23.444003 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:01:23.457 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:01:23.458759 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:01:33.564 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:01:33.565608 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:01:33.574 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:01:33.575998 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:01:43.686 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:01:43.687754 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:01:43.706 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:01:43.704740 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:01:53.835 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:01:53.836909 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:01:53.848 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:01:53.850035 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:02:03.934 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:02:03.935811 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:02:03.946 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:02:03.947780 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:02:14.050 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:02:14.051899 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:02:14.064 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:02:14.065667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:02:24.136 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:02:24.137278 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:02:24.149 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:02:24.151236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:02:34.274 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:02:34.276079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:02:34.294 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:02:34.296178 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:02:44.422 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:02:44.423282 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:02:44.433 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:02:44.434912 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:02:54.587 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:02:54.588585 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:02:54.638 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:02:54.639986 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:03:04.819 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:03:04.820824 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:03:04.832 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:03:04.833786 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:03:14.970 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:03:14.971753 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:03:14.980 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:03:14.982029 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:03:25.094 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:03:25.095130 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:03:25.108 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:03:25.110073 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:03:35.186 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:03:35.187357 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:03:35.199 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:03:35.200562 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:03:45.284 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:03:45.285431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:03:45.297 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:03:45.298692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:03:55.441 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:03:55.442870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:03:55.456 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:03:55.457224 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:04:05.560 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:04:05.561792 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:04:05.578 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:04:05.579519 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:04:15.676 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:04:15.677229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:04:15.689 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:04:15.690957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:04:25.789 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:04:25.790127 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:04:25.805 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:04:25.806328 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:04:35.963 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:04:35.964177 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:04:35.976 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:04:35.978074 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:04:46.091 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:04:46.091695 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:04:46.108 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:04:46.109273 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:04:56.223 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:04:56.224791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:04:56.234 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:04:56.235401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:05:06.345 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:05:06.346813 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:05:06.360 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:05:06.361558 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:05:16.469 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:05:16.470268 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:05:16.483 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:05:16.485064 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:05:26.553 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:05:26.552527 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:05:26.604 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:05:26.605272 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:05:36.708 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:05:36.709659 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:05:36.721 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:05:36.722207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:05:46.824 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:05:46.825308 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:05:46.840 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:05:46.842002 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:05:56.943 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:05:56.943919 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:05:56.956 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:05:56.957390 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:06:07.055 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:06:07.056140 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:06:07.065 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:06:07.066519 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:06:17.186 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:06:17.187475 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:06:17.197 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:06:17.198510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:06:27.317 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:06:27.318727 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:06:27.330 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:06:27.331890 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:06:37.443 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:06:37.444264 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:06:37.456 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:06:37.457105 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:06:47.524 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:06:47.525894 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:06:47.536 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:06:47.537910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:06:57.601 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:06:57.602561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:06:57.617 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:06:57.619290 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:07:07.719 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:07:07.720376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:07:07.733 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:07:07.734402 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:07:17.804 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:07:17.806032 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:07:17.818 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:07:17.820187 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:07:27.933 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:07:27.934153 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:07:27.944 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:07:27.945855 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:07:38.017 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:07:38.018574 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:07:38.029 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:07:38.030385 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:07:48.106 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:07:48.107023 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:07:48.121 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:07:48.122455 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:07:58.238 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:07:58.239186 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:07:58.253 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:07:58.254856 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:08:08.348 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:08:08.348297 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:08:08.362 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:08:08.363492 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:08:18.446 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:08:18.447056 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:08:18.460 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:08:18.461362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:08:28.532 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:08:28.533466 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:08:28.544 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:08:28.546107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:08:38.649 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:08:38.650216 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:08:38.663 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:08:38.664410 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:08:48.732 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:08:48.733964 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:08:48.807 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:08:48.805119 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:08:58.885 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:08:58.886417 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:08:58.897 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:08:58.899001 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:09:09.007 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:09:09.008805 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:09:09.024 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:09:09.025858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:09:19.136 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:09:19.137057 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:09:19.146 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:09:19.147181 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:09:29.277 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:09:29.278133 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:09:29.289 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:09:29.290265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:09:39.409 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:09:39.410670 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:09:39.421 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:09:39.422249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:09:49.492 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:09:49.493526 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:09:49.504 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:09:49.505288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:09:59.615 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:09:59.616461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:09:59.627 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:09:59.628857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:10:09.763 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:10:09.764519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:10:09.777 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:10:09.778078 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:10:19.841 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:10:19.839035 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:10:19.851 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:10:19.852447 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:10:29.917 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:10:29.918315 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:10:29.983 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:10:29.983978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:10:40.067 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:10:40.068463 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:10:40.080 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:10:40.081773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:10:50.227 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:10:50.228253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:10:50.239 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:10:50.241024 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:11:00.317 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:11:00.316214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:11:00.329 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:11:00.330185 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:11:10.426 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:11:10.427305 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:11:10.440 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:11:10.441486 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:11:20.558 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:11:20.559219 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:11:20.580 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:11:20.580409 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:11:30.702 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:11:30.703543 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:11:30.713 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:11:30.714486 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:11:40.857 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:11:40.857881 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:11:40.869 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:11:40.870575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:11:51.012 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:11:51.012629 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:11:51.023 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:11:51.025072 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:12:01.135 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:12:01.135834 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:12:01.146 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:12:01.147727 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:12:11.214 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:12:11.215015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:12:11.225 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:12:11.227184 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:12:21.294 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:12:21.294927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:12:21.305 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:12:21.306856 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:12:31.383 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:12:31.383950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:12:31.443 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:12:31.444321 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:12:41.541 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:12:41.542240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:12:41.618 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:12:41.619612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:12:51.695 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:12:51.696101 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:12:51.750 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:12:51.751838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:12:58.174 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:12:58.174853 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T16:12:58.854 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:12:58.854916 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T16:12:58.844+0000 7efdba2ff380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T16:12:58.854 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T16:12:58.844+0000 7efdba2ff380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T16:12:58.855 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T16:12:58.844+0000 7efdba2ff380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T16:12:58.855 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T16:13:01.843 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:13:01.844361 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:13:01.875 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:13:01.871609 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:13:11.942 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:13:11.942976 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:13:11.954 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:13:11.955764 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:13:22.032 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:13:22.027949 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:13:22.099 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:13:22.099944 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:13:32.221 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:13:32.222177 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:13:32.232 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:13:32.233329 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:13:42.382 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:13:42.382850 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:13:42.393 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:13:42.394711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:13:52.570 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:13:52.570925 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:13:52.584 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:13:52.584993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:14:02.710 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:14:02.711352 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:14:02.967 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:14:02.720950 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:14:12.842 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:14:12.843434 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:14:12.859 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:14:12.860535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:14:22.970 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:14:22.971074 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:14:22.982 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:14:22.983946 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:14:33.076 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:14:33.077234 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:14:33.131 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:14:33.131969 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:14:43.236 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:14:43.237137 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:14:43.248 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:14:43.249594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:14:53.340 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:14:53.341390 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:14:53.351 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:14:53.352143 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:15:03.482 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:15:03.483550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:15:03.494 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:15:03.495254 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:15:13.584 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:15:13.584135 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:15:13.598 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:15:13.599212 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:15:23.657 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:15:23.657053 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:15:23.670 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:15:23.671803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:15:33.753 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:15:33.754145 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:15:33.776 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:15:33.776790 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:15:43.897 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:15:43.897483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:15:43.912 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:15:43.913544 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:15:54.020 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:15:54.021118 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:15:54.032 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:15:54.033303 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:16:04.096 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:16:04.097260 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:16:04.179 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:16:04.180231 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:16:14.309 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:16:14.310210 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:16:14.327 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:16:14.325175 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:16:24.433 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:16:24.434085 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:16:24.446 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:16:24.446778 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:16:34.512 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:16:34.513346 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:16:34.576 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:16:34.577659 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:16:44.637 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:16:44.638025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:16:44.650 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:16:44.651482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:16:54.763 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:16:54.763415 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:16:54.777 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:16:54.778147 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:17:04.902 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:17:04.903370 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:17:04.916 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:17:04.916936 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:17:15.011 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:17:15.012112 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:17:15.024 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:17:15.025640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:17:25.105 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:17:25.105583 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:17:25.116 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:17:25.116832 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:17:35.235 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:17:35.236326 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:17:35.249 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:17:35.249920 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:17:45.354 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:17:45.354373 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:17:45.367 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:17:45.368433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:17:55.426 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:17:55.427120 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:17:55.440 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:17:55.441258 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:18:05.563 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:18:05.563644 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:18:05.576 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:18:05.576663 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:18:15.707 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:18:15.707509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:18:15.720 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:18:15.721372 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:18:25.791 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:18:25.792231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:18:25.843 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:18:25.844122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:18:35.929 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:18:35.929990 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:18:35.950 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:18:35.950763 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:18:46.060 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:18:46.060378 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:18:46.073 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:18:46.074551 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:18:56.198 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:18:56.197710 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:18:56.209 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:18:56.210126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:19:06.381 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:19:06.381998 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:19:06.396 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:19:06.397145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:19:16.462 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:19:16.463463 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:19:16.477 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:19:16.477725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:19:26.547 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:19:26.547866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:19:26.562 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:19:26.562429 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:19:36.679 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:19:36.680257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:19:36.695 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:19:36.695725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:19:46.755 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:19:46.755448 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:19:46.768 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:19:46.769591 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:19:56.853 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:19:56.853706 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:19:56.867 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:19:56.867935 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:20:06.930 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:20:06.931240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:20:06.943 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:20:06.943207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:20:17.087 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:20:17.087760 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:20:17.103 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:20:17.104395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:20:27.208 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:20:27.208502 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:20:27.219 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:20:27.220338 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:20:37.286 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:20:37.286930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:20:37.299 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:20:37.300420 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:20:47.371 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:20:47.371412 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:20:47.420 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:20:47.421351 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:20:57.504 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:20:57.504727 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:20:57.516 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:20:57.517185 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:21:07.600 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:21:07.601225 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:21:07.614 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:21:07.614642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:21:17.685 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:21:17.685723 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:21:17.697 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:21:17.698032 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:21:27.781 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:21:27.781512 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:21:27.796 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:21:27.796620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:21:37.859 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:21:37.859659 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:21:37.872 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:21:37.872810 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:21:47.995 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:21:47.996236 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:21:48.009 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:21:48.009895 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:21:58.179 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:21:58.180243 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:21:58.193 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:21:58.194066 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:22:08.274 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:22:08.275049 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:22:08.288 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:22:08.288596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:22:18.482 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:22:18.482930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:22:18.495 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:22:18.496312 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:22:28.603 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:22:28.603195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:22:28.615 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:22:28.616711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:22:38.737 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:22:38.737309 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:22:38.749 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:22:38.749511 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:22:48.878 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:22:48.878948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:22:48.892 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:22:48.892905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:22:59.021 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:22:59.021446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:22:59.033 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:22:59.033661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:23:09.111 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:23:09.111519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:23:09.165 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:23:09.165948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:23:19.275 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:23:19.275166 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:23:19.289 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:23:19.290266 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:23:29.385 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:23:29.385374 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:23:29.401 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:23:29.401672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:23:39.521 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:23:39.521676 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:23:39.532 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:23:39.532747 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:23:49.603 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:23:49.604216 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:23:49.616 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:23:49.617338 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:23:59.690 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:23:59.691075 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:23:59.701 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:23:59.701938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:24:09.766 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:24:09.767130 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:24:09.781 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:24:09.782394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:24:19.916 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:24:19.917240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:24:19.929 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:24:19.930081 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:24:30.040 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:24:30.040836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:24:30.053 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:24:30.053936 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:24:40.170 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:24:40.170789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:24:40.183 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:24:40.184200 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:24:50.299 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:24:50.299475 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:24:50.315 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:24:50.315887 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:25:00.423 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:25:00.424291 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:25:00.451 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:25:00.452137 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:25:10.549 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:25:10.550187 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:25:10.564 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:25:10.564847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:25:20.673 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:25:20.674257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:25:20.688 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:25:20.689183 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:25:30.790 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:25:30.791057 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:25:30.857 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:25:30.858141 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:25:40.965 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:25:40.966288 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:25:40.978 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:25:40.979787 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:25:51.160 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:25:51.161065 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:25:51.204 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:25:51.205608 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:26:01.348 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:26:01.349234 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:26:01.382 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:26:01.383023 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:26:11.509 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:26:11.508636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:26:11.521 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:26:11.522403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:26:21.637 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:26:21.637124 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:26:21.647 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:26:21.647947 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:26:31.724 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:26:31.725183 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:26:31.769 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:26:31.770326 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:26:41.840 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:26:41.840943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:26:41.854 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:26:41.855316 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:26:51.991 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:26:51.992211 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:26:52.006 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:26:52.006774 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:27:02.095 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:27:02.095388 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:27:02.109 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:27:02.108040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:27:12.219 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:27:12.220231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:27:12.230 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:27:12.231389 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:27:22.299 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:27:22.300091 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:27:22.313 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:27:22.313559 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:27:32.434 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:27:32.434481 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:27:32.448 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:27:32.448860 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:27:42.519 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:27:42.519613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:27:42.532 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:27:42.533061 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:27:52.615 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:27:52.615555 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:27:52.680 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:27:52.681336 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:28:02.798 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:28:02.799409 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:28:02.808 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:28:02.808971 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:28:12.919 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:28:12.919258 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:28:12.932 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:28:12.932762 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:28:22.996 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:28:22.996856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:28:23.012 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:28:23.013330 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:28:33.080 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:28:33.081625 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:28:33.094 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:28:33.094792 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:28:43.242 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:28:43.242606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:28:43.253 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:28:43.254407 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:28:53.358 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:28:53.358821 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:28:53.370 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:28:53.371422 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:29:03.484 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:29:03.485305 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:29:03.496 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:29:03.497314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:29:13.618 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:29:13.619302 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:29:13.632 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:29:13.633650 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:29:23.734 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:29:23.734654 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:29:23.748 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:29:23.749058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:29:33.840 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:29:33.840550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:29:33.856 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:29:33.857197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:29:38.868 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:29:38.869208 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T16:29:39.510 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:29:39.510910 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T16:29:39.501+0000 7f62c2a18380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T16:29:39.511 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T16:29:39.501+0000 7f62c2a18380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T16:29:39.511 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T16:29:39.501+0000 7f62c2a18380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T16:29:39.511 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T16:29:44.009 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:29:44.009445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:29:44.021 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:29:44.022009 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:29:54.153 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:29:54.153662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:29:54.167 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:29:54.167688 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:30:04.236 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:30:04.236754 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:30:04.250 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:30:04.250742 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:30:14.371 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:30:14.371553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:30:14.383 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:30:14.384623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:30:24.477 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:30:24.477679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:30:24.488 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:30:24.489249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:30:34.583 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:30:34.584415 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:30:34.595 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:30:34.596309 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:30:44.714 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:30:44.715122 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:30:44.724 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:30:44.725180 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:30:54.783 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:30:54.783764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:30:54.797 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:30:54.798500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:31:04.894 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:31:04.895070 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:31:04.903 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:31:04.904241 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:31:15.069 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:31:15.070276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:31:15.081 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:31:15.081901 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:31:25.189 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:31:25.189666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:31:25.206 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:31:25.206802 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:31:35.327 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:31:35.327952 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:31:35.340 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:31:35.340864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:31:45.422 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:31:45.423229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:31:45.435 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:31:45.436380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:31:55.510 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:31:55.510397 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:31:55.522 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:31:55.522841 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:32:05.590 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:32:05.590463 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:32:05.604 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:32:05.605067 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:32:15.759 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:32:15.759603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:32:15.774 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:32:15.775284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:32:25.914 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:32:25.914943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:32:25.925 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:32:25.926489 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:32:36.070 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:32:36.070769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:32:36.086 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:32:36.086765 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:32:46.176 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:32:46.176857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:32:46.191 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:32:46.192127 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:32:56.261 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:32:56.261995 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:32:56.275 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:32:56.276271 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:33:06.375 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:33:06.375687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:33:06.388 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:33:06.388543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:33:16.535 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:33:16.534889 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:33:16.554 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:33:16.550542 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:33:26.698 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:33:26.699295 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:33:26.713 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:33:26.714499 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:33:36.848 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:33:36.849000 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:33:36.861 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:33:36.861505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:33:46.989 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:33:46.989469 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:33:47.001 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:33:47.002046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:33:57.098 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:33:57.098587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:33:57.139 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:33:57.115574 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:34:07.252 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:34:07.252288 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:34:07.264 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:34:07.265142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:34:17.429 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:34:17.429245 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:34:17.442 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:34:17.443040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:34:27.539 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:34:27.540099 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:34:27.552 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:34:27.552857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:34:37.645 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:34:37.645851 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:34:37.658 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:34:37.658506 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:34:47.768 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:34:47.768778 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:34:47.781 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:34:47.781783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:34:57.937 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:34:57.937593 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:34:57.955 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:34:57.956014 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:35:08.079 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:35:08.079018 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:35:08.091 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:35:08.091961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:35:18.162 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:35:18.162327 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:35:18.177 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:35:18.178040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:35:28.311 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:35:28.311519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:35:28.327 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:35:28.327348 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:35:38.439 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:35:38.440083 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:35:38.452 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:35:38.453407 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:35:48.532 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:35:48.533106 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:35:48.548 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:35:48.548192 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:35:58.655 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:35:58.655273 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:35:58.668 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:35:58.668255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:36:08.777 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:36:08.777339 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:36:08.791 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:36:08.792039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:36:18.873 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:36:18.874066 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:36:18.888 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:36:18.888983 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:36:29.028 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:36:29.028279 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:36:29.043 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:36:29.043943 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:36:39.152 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:36:39.152378 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:36:39.164 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:36:39.164800 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:36:49.314 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:36:49.314133 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:36:49.327 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:36:49.327612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:36:59.453 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:36:59.453499 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:36:59.468 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:36:59.468596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:37:09.617 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:37:09.617925 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:37:09.632 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:37:09.632299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:37:19.842 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:37:19.842048 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:37:19.856 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:37:19.856255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:37:29.987 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:37:29.987452 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:37:30.000 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:37:30.000936 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:37:40.106 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:37:40.106453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:37:40.120 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:37:40.120723 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:37:50.246 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:37:50.246224 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:37:50.260 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:37:50.260467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:38:00.336 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:38:00.336406 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:38:00.361 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:38:00.361230 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:38:10.478 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:38:10.478649 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:38:10.492 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:38:10.492548 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:38:20.638 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:38:20.638167 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:38:20.650 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:38:20.650579 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:38:30.719 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:38:30.719677 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:38:30.730 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:38:30.731134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:38:40.836 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:38:40.836501 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:38:40.849 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:38:40.849506 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:38:50.946 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:38:50.947180 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:38:50.957 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:38:50.957962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:39:01.038 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:39:01.039062 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:39:01.053 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:39:01.054299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:39:11.121 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:39:11.121604 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:39:11.133 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:39:11.134199 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:39:21.242 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:39:21.242872 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:39:21.255 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:39:21.255663 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:39:31.342 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:39:31.343158 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:39:31.353 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:39:31.354121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:39:41.463 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:39:41.462957 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:39:41.482 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:39:41.482152 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:39:51.623 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:39:51.624186 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:39:51.637 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:39:51.638068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:40:01.815 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:40:01.815241 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:40:01.830 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:40:01.830917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:40:11.962 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:40:11.961525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:40:11.973 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:40:11.973391 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:40:22.059 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:40:22.059572 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:40:22.072 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:40:22.072435 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:40:32.151 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:40:32.151206 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:40:32.182 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:40:32.179672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:40:42.297 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:40:42.298138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:40:42.309 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:40:42.310368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:40:52.373 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:40:52.374015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:40:52.388 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:40:52.389161 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:41:02.509 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:41:02.509446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:41:02.523 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:41:02.523443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:41:12.588 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:41:12.588391 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:41:12.602 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:41:12.603250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:41:22.692 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:41:22.692242 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:41:22.740 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:41:22.740504 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:41:32.842 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:41:32.842527 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:41:32.856 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:41:32.856525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:41:42.972 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:41:42.972858 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:41:42.989 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:41:42.986049 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:41:53.104 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:41:53.104328 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:41:53.120 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:41:53.120677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:42:03.232 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:42:03.233073 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:42:03.251 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:42:03.251715 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:42:13.367 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:42:13.367949 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:42:13.382 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:42:13.383336 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:42:23.510 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:42:23.510907 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:42:23.524 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:42:23.524603 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:42:33.589 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:42:33.589756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:42:33.603 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:42:33.604313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:42:43.677 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:42:43.677921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:42:43.768 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:42:43.768575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:42:53.856 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:42:53.856516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:42:53.938 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:42:53.938701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:43:03.996 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:43:03.997231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:43:04.007 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:43:04.007849 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:43:14.137 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:43:14.137232 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:43:14.152 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:43:14.152963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:43:24.222 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:43:24.222802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:43:24.237 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:43:24.237755 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:43:34.299 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:43:34.299493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:43:34.316 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:43:34.317426 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:43:44.409 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:43:44.409307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:43:44.471 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:43:44.471661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:43:54.534 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:43:54.531225 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:43:54.542 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:43:54.543189 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:44:04.659 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:44:04.659808 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:44:04.672 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:44:04.673227 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:44:14.797 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:44:14.797918 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:44:14.810 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:44:14.811244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:44:24.881 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:44:24.882142 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:44:24.893 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:44:24.894354 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:44:35.007 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:44:35.007895 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:44:35.019 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:44:35.020082 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:44:45.135 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:44:45.135682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:44:45.148 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:44:45.149022 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:44:55.259 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:44:55.259557 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:44:55.274 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:44:55.274580 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:45:05.389 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:45:05.389597 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:45:05.404 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:45:05.405225 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:45:15.477 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:45:15.478151 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:45:15.490 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:45:15.490655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:45:25.558 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:45:25.558955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:45:25.577 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:45:25.575528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:45:35.646 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:45:35.646470 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:45:35.659 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:45:35.660163 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:45:45.739 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:45:45.740344 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:45:45.752 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:45:45.752855 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:45:55.856 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:45:55.856926 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:45:55.869 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:45:55.870603 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:46:06.060 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:46:06.060454 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:46:06.073 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:46:06.074496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:46:16.222 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:46:16.223271 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:46:16.237 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:46:16.238124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:46:19.547 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:46:19.548011 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T16:46:20.209 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:46:20.209500 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T16:46:20.201+0000 7f4bee77c380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T16:46:20.210 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T16:46:20.201+0000 7f4bee77c380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T16:46:20.210 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T16:46:20.201+0000 7f4bee77c380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T16:46:20.210 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T16:46:26.367 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:46:26.367614 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:46:26.386 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:46:26.386943 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:46:36.677 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:46:36.678506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:46:36.690 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:46:36.690791 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:46:46.817 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:46:46.817588 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:46:46.832 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:46:46.832709 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:46:56.912 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:46:56.912812 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:46:56.923 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:46:56.923963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:47:07.054 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:47:07.054275 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:47:07.066 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:47:07.067228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:47:17.184 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:47:17.184235 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:47:17.196 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:47:17.197337 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:47:27.280 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:47:27.280902 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:47:27.297 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:47:27.297595 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:47:37.412 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:47:37.412874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:47:37.427 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:47:37.426087 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:47:47.529 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:47:47.530195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:47:47.584 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:47:47.585283 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:47:57.691 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:47:57.692302 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:47:57.746 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:47:57.746525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:48:07.878 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:48:07.879011 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:48:07.891 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:48:07.892193 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:48:17.993 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:48:17.994074 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:48:18.004 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:48:18.005285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:48:28.083 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:48:28.084455 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:48:28.098 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:48:28.098783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:48:38.214 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:48:38.214721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:48:38.227 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:48:38.228665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:48:48.294 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:48:48.294899 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:48:48.307 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:48:48.307669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:48:58.374 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:48:58.375015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:48:58.387 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:48:58.388447 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:49:08.513 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:49:08.508461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:49:08.523 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:49:08.524066 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:49:18.600 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:49:18.601181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:49:18.612 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:49:18.612768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:49:28.715 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:49:28.716275 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:49:28.740 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:49:28.741133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:49:38.837 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:49:38.837986 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:49:38.852 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:49:38.851730 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:49:49.014 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:49:49.015293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:49:49.026 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:49:49.027066 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:49:59.105 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:49:59.106089 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:49:59.118 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:49:59.118745 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:50:09.238 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:50:09.238834 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:50:09.251 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:50:09.252226 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:50:19.351 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:50:19.351693 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:50:19.366 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:50:19.367421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:50:29.482 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:50:29.482730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:50:29.493 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:50:29.494111 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:50:39.562 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:50:39.562384 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:50:39.619 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:50:39.619786 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:50:49.725 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:50:49.725953 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:50:49.756 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:50:49.757770 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:50:59.827 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:50:59.827872 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:50:59.841 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:50:59.842597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:51:09.968 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:51:09.969694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:51:09.984 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:51:09.985012 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:51:20.088 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:51:20.089052 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:51:20.101 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:51:20.102714 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:51:30.218 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:51:30.219085 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:51:30.231 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:51:30.232456 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:51:40.314 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:51:40.315181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:51:40.328 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:51:40.329295 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:51:50.427 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:51:50.428455 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:51:50.441 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:51:50.442158 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:52:00.527 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:52:00.528091 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:52:00.539 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:52:00.541011 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:52:10.658 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:52:10.659039 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:52:10.673 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:52:10.674437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:52:20.753 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:52:20.754090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:52:20.798 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:52:20.799325 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:52:30.917 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:52:30.917588 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:52:30.930 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:52:30.931566 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:52:40.998 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:52:40.999334 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:52:41.009 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:52:41.010230 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:52:51.074 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:52:51.075468 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:52:51.087 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:52:51.088632 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:53:01.193 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:53:01.193658 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:53:01.204 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:53:01.205865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:53:11.342 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:53:11.342918 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:53:11.358 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:53:11.359549 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:53:21.532 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:53:21.533313 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:53:21.546 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:53:21.547394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:53:31.658 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:53:31.659320 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:53:31.673 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:53:31.674404 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:53:41.791 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:53:41.792584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:53:41.804 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:53:41.805942 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:53:51.872 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:53:51.872843 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:53:51.885 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:53:51.886776 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:54:02.030 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:54:02.031269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:54:02.049 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:54:02.050241 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:54:12.117 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:54:12.118479 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:54:12.179 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:54:12.179782 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:54:22.309 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:54:22.309813 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:54:22.319 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:54:22.320449 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:54:32.461 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:54:32.462552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:54:32.474 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:54:32.475768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:54:42.600 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:54:42.600950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:54:42.611 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:54:42.613258 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:54:52.685 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:54:52.686772 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:54:52.729 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:54:52.730234 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:55:02.795 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:55:02.795906 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:55:02.809 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:55:02.810689 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:55:12.880 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:55:12.881053 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:55:12.891 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:55:12.892543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:55:22.984 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:55:22.985903 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:55:22.997 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:55:22.998829 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:55:33.073 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:55:33.074238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:55:33.087 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:55:33.088150 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:55:43.208 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:55:43.209935 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:55:43.220 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:55:43.221703 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:55:53.342 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:55:53.343214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:55:53.355 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:55:53.356571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:56:03.495 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:56:03.496127 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:56:03.506 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:56:03.507821 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:56:13.580 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:56:13.581106 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:56:13.655 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:56:13.655732 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:56:23.813 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:56:23.814752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:56:23.830 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:56:23.831571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:56:33.904 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:56:33.905204 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:56:33.918 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:56:33.919617 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:56:44.043 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:56:44.044504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:56:44.057 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:56:44.058264 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:56:54.170 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:56:54.170115 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:56:54.181 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:56:54.182736 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:57:04.303 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:57:04.302705 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:57:04.430 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:57:04.317043 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:57:14.389 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:57:14.390254 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:57:14.401 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:57:14.402857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:57:24.509 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:57:24.509988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:57:24.521 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:57:24.522721 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:57:34.634 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:57:34.635625 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:57:34.647 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:57:34.648486 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:57:44.784 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:57:44.785623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:57:44.799 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:57:44.800786 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:57:54.933 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:57:54.934260 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:57:54.947 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:57:54.948930 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:58:05.067 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:58:05.068351 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:58:05.080 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:58:05.081306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:58:15.162 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:58:15.164098 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:58:15.173 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:58:15.174442 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:58:25.292 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:58:25.293084 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:58:25.305 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:58:25.306875 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:58:35.445 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:58:35.446737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:58:35.460 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:58:35.461884 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:58:45.568 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:58:45.570046 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:58:45.581 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:58:45.583345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:58:55.688 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:58:55.689489 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:58:55.776 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:58:55.777521 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:59:05.870 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:59:05.871087 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:59:05.914 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:59:05.915947 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:59:15.979 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:59:15.980219 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:59:15.992 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:59:15.994119 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:59:26.103 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:59:26.104995 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:59:26.116 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:59:26.117895 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:59:36.192 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:59:36.193562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:59:36.210 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:59:36.211545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:59:46.278 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:59:46.279299 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:59:46.290 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:59:46.291930 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:59:56.403 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:59:56.399250 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:59:56.419 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 16:59:56.420800 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:00:06.500 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:00:06.501766 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:00:06.513 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:00:06.514549 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:00:16.645 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:00:16.646467 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:00:16.659 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:00:16.660823 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:00:26.760 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:00:26.762048 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:00:26.775 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:00:26.776976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:00:36.850 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:00:36.851925 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:00:36.862 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:00:36.864198 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:00:46.931 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:00:46.933163 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:00:46.944 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:00:46.946353 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:00:57.023 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:00:57.023501 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:00:57.033 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:00:57.035070 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:01:07.161 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:01:07.162811 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:01:07.175 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:01:07.177443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:01:17.252 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:01:17.254191 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:01:17.263 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:01:17.265485 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:01:27.408 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:01:27.409570 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:01:27.422 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:01:27.424075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:01:37.561 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:01:37.562874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:01:37.574 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:01:37.575653 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:01:47.705 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:01:47.706710 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:01:47.720 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:01:47.722236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:01:57.806 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:01:57.807919 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:01:57.879 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:01:57.880766 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:02:07.951 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:02:07.953019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:02:07.963 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:02:07.965477 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:02:18.026 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:02:18.028089 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:02:18.038 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:02:18.039794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:02:28.128 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:02:28.130205 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:02:28.141 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:02:28.143443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:02:38.250 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:02:38.251393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:02:38.267 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:02:38.268631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:02:48.350 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:02:48.351750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:02:48.364 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:02:48.365919 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:02:58.435 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:02:58.437001 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:02:58.453 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:02:58.454652 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:03:00.274 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:03:00.276045 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T17:03:00.964 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:03:00.964571 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T17:03:00.955+0000 7fe236a9d380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T17:03:00.964 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T17:03:00.955+0000 7fe236a9d380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T17:03:00.965 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T17:03:00.955+0000 7fe236a9d380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T17:03:00.965 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T17:03:08.555 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:03:08.557051 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:03:08.569 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:03:08.570989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:03:18.674 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:03:18.675300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:03:18.686 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:03:18.687783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:03:28.762 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:03:28.762398 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:03:28.773 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:03:28.775016 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:03:38.891 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:03:38.892905 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:03:38.899 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:03:38.901267 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:03:48.962 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:03:48.963558 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:03:48.977 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:03:48.979200 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:03:59.107 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:03:59.108693 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:03:59.120 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:03:59.122068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:04:09.227 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:04:09.228654 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:04:09.240 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:04:09.241726 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:04:19.312 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:04:19.313483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:04:19.389 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:04:19.390474 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:04:29.512 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:04:29.513659 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:04:29.526 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:04:29.528329 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:04:39.633 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:04:39.634575 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:04:39.646 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:04:39.648316 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:04:49.724 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:04:49.726378 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:04:49.737 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:04:49.738407 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:04:59.810 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:04:59.811955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:04:59.856 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:04:59.857546 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:05:09.928 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:05:09.930066 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:05:10.003 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:05:10.004232 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:05:20.080 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:05:20.082358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:05:20.095 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:05:20.096501 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:05:30.160 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:05:30.162107 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:05:30.173 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:05:30.174510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:05:40.302 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:05:40.303460 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:05:40.316 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:05:40.318383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:05:50.459 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:05:50.460902 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:05:50.475 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:05:50.473461 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:06:00.601 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:06:00.602981 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:06:00.614 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:06:00.616580 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:06:10.683 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:06:10.685154 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:06:10.728 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:06:10.729728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:06:20.847 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:06:20.848716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:06:20.860 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:06:20.861560 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:06:30.960 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:06:30.961452 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:06:30.974 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:06:30.975617 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:06:41.083 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:06:41.085171 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:06:41.094 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:06:41.096047 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:06:51.193 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:06:51.194208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:06:51.207 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:06:51.208988 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:07:01.309 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:07:01.310644 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:07:01.323 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:07:01.324837 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:07:11.383 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:07:11.384198 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:07:11.439 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:07:11.440478 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:07:21.520 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:07:21.521520 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:07:21.530 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:07:21.531957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:07:31.610 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:07:31.612231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:07:31.623 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:07:31.624679 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:07:41.746 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:07:41.747186 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:07:41.758 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:07:41.759674 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:07:51.834 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:07:51.835976 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:07:51.849 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:07:51.851460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:08:01.926 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:08:01.927374 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:08:01.941 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:08:01.943016 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:08:12.008 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:08:12.008476 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:08:12.022 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:08:12.024160 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:08:22.091 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:08:22.092981 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:08:22.105 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:08:22.107005 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:08:32.200 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:08:32.202052 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:08:32.212 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:08:32.214328 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:08:42.337 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:08:42.338843 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:08:42.719 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:08:42.350969 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:08:52.409 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:08:52.409027 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:08:52.452 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:08:52.453142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:09:02.530 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:09:02.531549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:09:02.544 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:09:02.546121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:09:12.650 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:09:12.651716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:09:12.663 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:09:12.664943 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:09:22.738 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:09:22.738029 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:09:22.751 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:09:22.752631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:09:32.868 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:09:32.869759 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:09:32.879 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:09:32.881274 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:09:42.985 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:09:42.986287 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:09:42.999 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:09:43.001273 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:09:53.106 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:09:53.107089 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:09:53.118 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:09:53.119756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:10:03.277 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:10:03.278743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:10:03.291 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:10:03.292985 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:10:13.357 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:10:13.358634 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:10:13.372 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:10:13.373588 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:10:23.510 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:10:23.510600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:10:23.541 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:10:23.542665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:10:33.611 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:10:33.612613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:10:33.678 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:10:33.679878 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:10:43.759 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:10:43.760268 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:10:43.772 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:10:43.774021 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:10:53.867 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:10:53.868963 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:10:53.881 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:10:53.882644 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:11:03.983 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:11:03.985052 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:11:03.995 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:11:03.997146 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:11:14.064 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:11:14.065521 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:11:14.077 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:11:14.079111 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:11:24.142 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:11:24.142431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:11:24.153 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:11:24.154924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:11:34.263 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:11:34.264876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:11:34.282 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:11:34.283562 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:11:44.405 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:11:44.406969 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:11:44.419 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:11:44.421072 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:11:54.494 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:11:54.495533 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:11:54.507 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:11:54.509280 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:12:04.620 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:12:04.621532 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:12:04.633 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:12:04.635000 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:12:14.700 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:12:14.701397 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:12:14.713 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:12:14.714777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:12:24.779 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:12:24.780843 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:12:24.790 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:12:24.792049 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:12:34.901 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:12:34.902289 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:12:34.918 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:12:34.919487 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:12:45.003 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:12:45.004064 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:12:45.015 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:12:45.017082 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:12:55.134 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:12:55.135158 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:12:55.148 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:12:55.149413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:13:05.256 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:13:05.257587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:13:05.270 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:13:05.271550 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:13:15.400 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:13:15.401591 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:13:15.415 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:13:15.416836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:13:25.485 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:13:25.486232 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:13:25.496 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:13:25.498253 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:13:35.610 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:13:35.611748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:13:35.624 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:13:35.625333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:13:45.740 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:13:45.740722 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:13:45.753 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:13:45.754424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:13:55.872 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:13:55.872306 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:13:55.883 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:13:55.884927 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:14:05.956 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:14:05.957741 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:14:05.966 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:14:05.967507 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:14:16.021 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:14:16.022400 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:14:16.033 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:14:16.035089 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:14:26.282 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:14:26.283373 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:14:26.296 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:14:26.297492 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:14:36.465 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:14:36.466906 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:14:36.479 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:14:36.480437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:14:46.615 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:14:46.616691 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:14:46.629 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:14:46.630306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:14:56.760 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:14:56.761776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:14:56.772 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:14:56.773978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:15:06.882 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:15:06.882285 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:15:06.894 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:15:06.895467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:15:16.967 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:15:16.968624 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:15:16.981 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:15:16.983179 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:15:27.053 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:15:27.054257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:15:27.266 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:15:27.068066 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:15:37.182 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:15:37.183237 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:15:37.212 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:15:37.212148 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:15:47.332 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:15:47.333791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:15:47.347 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:15:47.348497 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:15:57.446 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:15:57.447208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:15:57.460 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:15:57.461533 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:16:07.548 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:16:07.549184 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:16:07.564 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:16:07.565992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:16:17.634 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:16:17.635411 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:16:17.643 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:16:17.645001 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:16:27.754 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:16:27.755487 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:16:27.768 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:16:27.770068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:16:37.845 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:16:37.847009 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:16:37.904 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:16:37.905591 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:16:48.058 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:16:48.059876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:16:48.070 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:16:48.071583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:16:58.178 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:16:58.179474 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:16:58.187 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:16:58.188696 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:17:08.260 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:17:08.261890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:17:08.274 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:17:08.275618 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:17:18.344 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:17:18.344701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:17:18.355 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:17:18.357244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:17:28.463 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:17:28.463937 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:17:28.476 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:17:28.477511 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:17:38.579 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:17:38.580132 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:17:38.598 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:17:38.599927 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:17:48.708 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:17:48.709477 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:17:48.723 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:17:48.724312 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:17:58.808 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:17:58.810201 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:17:58.825 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:17:58.826504 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:18:08.925 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:18:08.926429 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:18:08.938 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:18:08.940009 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:18:19.044 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:18:19.045557 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:18:19.058 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:18:19.059785 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:18:29.133 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:18:29.134407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:18:29.147 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:18:29.148720 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:18:39.287 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:18:39.289087 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:18:39.299 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:18:39.301160 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:18:49.377 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:18:49.378329 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:18:49.391 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:18:49.393468 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:18:59.508 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:18:59.509936 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:18:59.521 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:18:59.523252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:19:09.642 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:19:09.644124 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:19:09.656 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:19:09.658257 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:19:19.783 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:19:19.784675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:19:19.795 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:19:19.796765 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:19:29.907 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:19:29.908379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:19:29.928 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:19:29.929935 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:19:39.995 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:19:39.996835 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:19:40.061 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:19:40.062171 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:19:40.972 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:19:40.973657 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T17:19:41.609 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:19:41.610132 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T17:19:41.599+0000 7fe6800c2380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T17:19:41.609 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T17:19:41.599+0000 7fe6800c2380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T17:19:41.610 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T17:19:41.599+0000 7fe6800c2380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T17:19:41.610 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T17:19:50.178 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:19:50.179658 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:19:50.191 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:19:50.192795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:20:00.380 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:20:00.382020 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:20:00.394 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:20:00.396115 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:20:10.503 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:20:10.504393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:20:10.519 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:20:10.520835 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:20:20.590 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:20:20.591698 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:20:20.603 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:20:20.604616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:20:30.733 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:20:30.734584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:20:30.746 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:20:30.747888 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:20:40.908 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:20:40.909429 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:20:40.923 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:20:40.924319 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:20:51.008 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:20:51.008861 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:20:51.020 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:20:51.022560 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:21:01.129 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:21:01.129911 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:21:01.140 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:21:01.141797 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:21:11.197 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:21:11.199195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:21:11.210 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:21:11.212069 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:21:21.328 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:21:21.329590 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:21:21.341 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:21:21.342564 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:21:31.411 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:21:31.412261 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:21:31.428 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:21:31.426909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:21:41.549 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:21:41.550293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:21:41.561 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:21:41.563375 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:21:51.643 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:21:51.644291 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:21:51.655 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:21:51.656806 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:22:01.781 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:22:01.782525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:22:01.797 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:22:01.798642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:22:11.896 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:22:11.897695 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:22:11.909 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:22:11.910672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:22:21.981 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:22:21.983008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:22:22.038 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:22:22.040311 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:22:32.167 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:22:32.168265 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:22:32.178 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:22:32.180378 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:22:42.253 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:22:42.254090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:22:42.265 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:22:42.266579 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:22:52.405 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:22:52.407136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:22:52.420 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:22:52.422196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:23:02.540 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:23:02.541663 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:23:02.554 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:23:02.556292 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:23:12.672 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:23:12.673973 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:23:12.687 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:23:12.689313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:23:22.784 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:23:22.786101 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:23:22.800 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:23:22.801711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:23:32.917 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:23:32.918599 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:23:32.930 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:23:32.932179 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:23:43.043 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:23:43.044772 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:23:43.056 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:23:43.057748 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:23:53.166 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:23:53.167790 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:23:53.179 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:23:53.180844 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:24:03.275 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:24:03.276426 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:24:03.287 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:24:03.289140 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:24:13.369 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:24:13.370435 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:24:13.382 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:24:13.383862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:24:23.495 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:24:23.496352 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:24:23.509 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:24:23.510838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:24:33.583 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:24:33.584892 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:24:33.596 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:24:33.598610 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:24:43.667 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:24:43.667201 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:24:43.679 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:24:43.679967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:24:53.808 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:24:53.809866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:24:53.821 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:24:53.823348 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:25:03.942 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:25:03.943935 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:25:03.955 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:25:03.956634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:25:14.066 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:25:14.067660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:25:14.078 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:25:14.079741 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:25:24.149 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:25:24.150953 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:25:24.162 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:25:24.163971 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:25:34.228 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:25:34.229703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:25:34.241 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:25:34.242649 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:25:44.352 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:25:44.354180 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:25:44.365 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:25:44.366296 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:25:54.486 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:25:54.487948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:25:54.499 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:25:54.500824 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:26:04.612 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:26:04.613743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:26:04.635 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:26:04.635656 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:26:14.746 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:26:14.747604 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:26:14.760 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:26:14.762278 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:26:24.868 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:26:24.870105 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:26:24.883 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:26:24.882842 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:26:34.962 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:26:34.963400 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:26:34.977 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:26:34.979289 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:26:45.060 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:26:45.061895 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:26:45.074 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:26:45.075560 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:26:55.177 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:26:55.179170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:26:55.193 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:26:55.194862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:27:05.320 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:27:05.322224 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:27:05.336 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:27:05.338562 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:27:15.497 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:27:15.499235 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:27:16.115 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:27:15.510193 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:27:25.635 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:27:25.636564 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:27:25.647 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:27:25.649004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:27:35.757 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:27:35.758420 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:27:35.770 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:27:35.771730 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:27:45.841 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:27:45.842452 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:27:45.862 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:27:45.858834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:27:55.919 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:27:55.920984 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:27:55.932 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:27:55.934185 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:28:06.037 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:28:06.037802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:28:06.050 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:28:06.051917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:28:16.159 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:28:16.160131 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:28:16.173 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:28:16.175030 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:28:26.409 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:28:26.411200 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:28:26.423 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:28:26.425546 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:28:36.566 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:28:36.567898 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:28:36.575 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:28:36.577373 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:28:46.671 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:28:46.672786 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:28:46.684 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:28:46.686519 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:28:56.747 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:28:56.748871 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:28:56.759 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:28:56.760938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:29:06.869 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:29:06.871155 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:29:06.884 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:29:06.885564 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:29:17.002 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:29:17.003621 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:29:17.014 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:29:17.015924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:29:27.084 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:29:27.085523 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:29:27.099 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:29:27.100687 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:29:37.190 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:29:37.191992 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:29:37.203 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:29:37.204747 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:29:47.308 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:29:47.309483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:29:47.321 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:29:47.323306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:29:57.437 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:29:57.438886 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:29:57.446 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:29:57.448345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:30:07.552 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:30:07.553470 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:30:07.565 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:30:07.567047 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:30:17.721 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:30:17.722526 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:30:17.813 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:30:17.814739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:30:27.921 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:30:27.922764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:30:27.983 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:30:27.984567 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:30:38.092 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:30:38.094573 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:30:38.105 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:30:38.106820 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:30:48.168 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:30:48.169965 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:30:48.181 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:30:48.183111 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:30:58.255 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:30:58.256806 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:30:58.268 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:30:58.269747 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:31:08.389 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:31:08.390578 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:31:08.402 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:31:08.403660 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:31:18.502 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:31:18.503776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:31:18.519 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:31:18.520832 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:31:28.588 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:31:28.590251 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:31:28.604 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:31:28.606124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:31:38.714 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:31:38.714824 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:31:38.726 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:31:38.727992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:31:48.829 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:31:48.830781 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:31:48.842 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:31:48.843896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:31:58.959 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:31:58.961435 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:31:58.973 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:31:58.975565 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:32:09.094 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:32:09.095713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:32:09.107 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:32:09.109042 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:32:19.216 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:32:19.217713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:32:19.229 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:32:19.231156 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:32:29.353 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:32:29.355049 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:32:29.369 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:32:29.370932 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:32:39.449 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:32:39.450509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:32:39.464 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:32:39.466400 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:32:49.574 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:32:49.576191 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:32:49.586 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:32:49.588275 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:32:59.733 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:32:59.735367 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:32:59.744 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:32:59.746855 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:33:09.871 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:33:09.872594 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:33:09.882 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:33:09.884602 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:33:20.008 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:33:20.009445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:33:20.022 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:33:20.023979 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:33:30.124 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:33:30.126455 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:33:30.151 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:33:30.152773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:33:40.225 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:33:40.227063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:33:40.238 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:33:40.240448 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:33:50.310 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:33:50.310759 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:33:50.378 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:33:50.380148 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:34:00.454 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:34:00.456260 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:34:00.468 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:34:00.470752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:34:10.563 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:34:10.564430 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:34:10.578 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:34:10.580071 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:34:20.649 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:34:20.651425 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:34:20.663 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:34:20.664790 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:34:30.784 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:34:30.786355 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:34:30.799 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:34:30.800716 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:34:40.910 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:34:40.912350 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:34:40.922 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:34:40.924464 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:34:51.026 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:34:51.028288 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:34:51.038 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:34:51.040379 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:35:01.128 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:35:01.129796 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:35:01.138 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:35:01.140223 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:35:11.206 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:35:11.207098 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:35:11.218 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:35:11.220187 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:35:21.322 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:35:21.324007 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:35:21.337 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:35:21.338848 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:35:31.409 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:35:31.410681 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:35:31.467 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:35:31.468555 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:35:41.582 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:35:41.584010 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:35:41.594 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:35:41.596116 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:35:51.794 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:35:51.795651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:35:51.808 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:35:51.810373 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:36:01.908 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:36:01.909565 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:36:01.921 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:36:01.922803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:36:12.027 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:36:12.029037 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:36:12.041 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:36:12.043184 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:36:21.615 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:36:21.616719 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T17:36:22.104 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:36:22.106196 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:36:22.119 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:36:22.121029 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:36:22.259 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:36:22.260703 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T17:36:22.252+0000 7f59fd9e8380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T17:36:22.260 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T17:36:22.252+0000 7f59fd9e8380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T17:36:22.260 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T17:36:22.252+0000 7f59fd9e8380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T17:36:22.260 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T17:36:32.195 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:36:32.196815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:36:32.245 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:36:32.247340 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:36:42.339 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:36:42.340207 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:36:42.386 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:36:42.388353 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:36:52.460 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:36:52.461330 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:36:52.473 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:36:52.474833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:37:02.553 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:37:02.554791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:37:02.566 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:37:02.567961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:37:12.717 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:37:12.718809 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:37:12.735 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:37:12.737017 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:37:22.820 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:37:22.822157 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:37:22.839 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:37:22.840351 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:37:32.956 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:37:32.957809 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:37:32.970 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:37:32.972189 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:37:43.074 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:37:43.075698 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:37:43.087 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:37:43.088392 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:37:53.230 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:37:53.231743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:37:53.240 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:37:53.242449 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:38:03.306 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:38:03.307663 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:38:03.319 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:38:03.320608 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:38:13.393 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:38:13.395028 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:38:13.407 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:38:13.408895 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:38:23.480 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:38:23.482296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:38:23.489 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:38:23.491498 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:38:33.559 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:38:33.559255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:38:33.573 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:38:33.574621 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:38:43.679 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:38:43.679596 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:38:43.692 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:38:43.690714 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:38:53.768 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:38:53.769294 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:38:53.780 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:38:53.782279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:39:03.857 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:39:03.858048 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:39:03.921 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:39:03.923017 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:39:14.007 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:39:14.008596 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:39:14.021 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:39:14.022639 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:39:24.131 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:39:24.132755 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:39:24.142 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:39:24.143626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:39:34.241 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:39:34.242464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:39:34.254 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:39:34.255951 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:39:44.337 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:39:44.338291 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:39:44.349 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:39:44.350526 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:39:54.469 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:39:54.470632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:39:54.484 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:39:54.484393 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:40:04.610 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:40:04.611353 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:40:04.623 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:40:04.625208 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:40:14.726 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:40:14.727310 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:40:14.741 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:40:14.742509 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:40:24.857 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:40:24.858576 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:40:24.870 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:40:24.871699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:40:35.031 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:40:35.032098 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:40:35.042 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:40:35.044037 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:40:45.222 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:40:45.223232 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:40:45.235 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:40:45.236291 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:40:55.376 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:40:55.377196 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:40:55.389 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:40:55.390532 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:41:05.491 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:41:05.492014 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:41:05.501 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:41:05.503064 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:41:15.577 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:41:15.578119 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:41:15.590 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:41:15.591604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:41:25.698 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:41:25.699085 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:41:25.713 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:41:25.714210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:41:35.797 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:41:35.797979 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:41:35.811 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:41:35.812210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:41:45.883 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:41:45.884410 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:41:45.895 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:41:45.896679 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:41:56.010 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:41:56.011360 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:41:56.022 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:41:56.023432 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:42:06.099 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:42:06.099379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:42:06.110 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:42:06.111768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:42:16.217 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:42:16.218243 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:42:16.243 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:42:16.244052 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:42:26.331 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:42:26.331950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:42:26.345 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:42:26.346960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:42:36.518 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:42:36.519263 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:42:36.575 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:42:36.576468 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:42:46.679 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:42:46.680587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:42:46.691 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:42:46.692843 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:42:56.774 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:42:56.775221 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:42:56.844 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:42:56.845213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:43:06.997 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:43:06.998414 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:43:07.006 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:43:07.007543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:43:17.113 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:43:17.114593 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:43:17.125 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:43:17.126915 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:43:27.199 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:43:27.200144 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:43:27.271 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:43:27.272213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:43:37.359 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:43:37.360101 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:43:37.371 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:43:37.373101 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:43:47.444 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:43:47.444813 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:43:47.459 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:43:47.460284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:43:57.592 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:43:57.593198 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:43:57.605 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:43:57.606327 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:44:07.676 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:44:07.677035 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:44:07.690 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:44:07.691245 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:44:17.759 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:44:17.760792 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:44:17.775 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:44:17.776315 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:44:27.853 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:44:27.854004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:44:27.867 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:44:27.868089 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:44:37.928 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:44:37.929487 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:44:37.938 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:44:37.939930 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:44:48.101 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:44:48.101643 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:44:48.112 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:44:48.113575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:44:58.237 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:44:58.237700 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:44:58.252 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:44:58.252708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:45:08.332 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:45:08.332878 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:45:08.343 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:45:08.344693 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:45:18.418 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:45:18.419046 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:45:18.429 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:45:18.430626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:45:28.553 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:45:28.553690 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:45:28.566 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:45:28.567787 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:45:38.688 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:45:38.688142 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:45:38.702 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:45:38.703461 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:45:48.783 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:45:48.783732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:45:48.801 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:45:48.801979 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:45:58.953 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:45:58.953878 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:45:58.967 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:45:58.964658 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:46:09.089 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:46:09.090102 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:46:09.104 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:46:09.105375 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:46:19.210 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:46:19.210672 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:46:19.223 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:46:19.224454 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:46:29.333 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:46:29.333539 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:46:29.345 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:46:29.346634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:46:39.456 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:46:39.457478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:46:39.470 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:46:39.471217 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:46:49.556 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:46:49.557268 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:46:49.567 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:46:49.567921 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:46:59.687 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:46:59.688014 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:46:59.697 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:46:59.698556 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:47:09.803 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:47:09.803900 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:47:09.815 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:47:09.816245 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:47:19.912 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:47:19.912959 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:47:19.926 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:47:19.927748 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:47:29.995 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:47:29.995742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:47:30.010 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:47:30.011258 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:47:40.124 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:47:40.124598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:47:40.135 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:47:40.136559 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:47:50.237 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:47:50.238128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:47:50.252 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:47:50.252886 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:48:00.414 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:48:00.415773 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:48:00.426 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:48:00.427269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:48:10.590 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:48:10.590827 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:48:10.603 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:48:10.604141 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:48:20.723 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:48:20.723879 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:48:20.735 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:48:20.736444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:48:30.815 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:48:30.816392 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:48:30.833 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:48:30.830585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:48:40.897 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:48:40.898484 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:48:40.911 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:48:40.911731 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:48:51.016 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:48:51.016888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:48:51.025 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:48:51.026166 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:49:01.143 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:49:01.143951 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:49:01.151 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:49:01.152866 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:49:11.264 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:49:11.264172 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:49:11.277 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:49:11.278018 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:49:21.416 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:49:21.416581 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:49:21.435 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:49:21.435879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:49:31.513 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:49:31.513932 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:49:31.527 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:49:31.528738 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:49:41.625 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:49:41.626061 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:49:41.641 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:49:41.642236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:49:51.776 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:49:51.777140 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:49:51.790 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:49:51.790740 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:50:01.935 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:50:01.936082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:50:01.948 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:50:01.949531 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:50:12.140 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:50:12.140888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:50:12.153 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:50:12.154491 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:50:22.341 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:50:22.342271 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:50:22.356 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:50:22.357360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:50:32.451 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:50:32.452187 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:50:32.509 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:50:32.509783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:50:42.582 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:50:42.583152 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:50:42.781 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:50:42.782478 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:50:52.998 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:50:52.998896 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:50:53.010 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:50:53.011588 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:51:03.170 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:51:03.170689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:51:03.183 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:51:03.184050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:51:13.285 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:51:13.286235 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:51:13.297 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:51:13.298380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:51:23.375 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:51:23.376367 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:51:23.390 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:51:23.391454 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:51:33.503 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:51:33.503809 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:51:33.514 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:51:33.515599 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:51:43.584 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:51:43.585072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:51:43.596 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:51:43.597150 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:51:53.666 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:51:53.667113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:51:53.733 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:51:53.733712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:52:03.836 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:52:03.837041 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:52:03.848 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:52:03.849529 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:52:13.978 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:52:13.978610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:52:13.988 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:52:13.989476 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:52:24.152 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:52:24.153006 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:52:24.166 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:52:24.167149 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:52:34.266 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:52:34.266856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:52:34.279 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:52:34.280877 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:52:44.397 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:52:44.397434 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:52:44.411 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:52:44.412214 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:52:54.475 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:52:54.475480 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:52:54.488 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:52:54.489261 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:53:02.266 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:53:02.267286 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T17:53:02.916 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:53:02.916868 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T17:53:02.908+0000 7f65e9f4e380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T17:53:02.917 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T17:53:02.908+0000 7f65e9f4e380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T17:53:02.917 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T17:53:02.908+0000 7f65e9f4e380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T17:53:02.917 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T17:53:04.556 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:53:04.556923 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:53:04.567 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:53:04.568665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:53:14.688 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:53:14.688857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:53:14.699 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:53:14.700918 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:53:24.840 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:53:24.841215 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:53:24.853 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:53:24.854780 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:53:34.972 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:53:34.972803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:53:34.985 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:53:34.986439 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:53:45.067 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:53:45.067731 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:53:45.080 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:53:45.080908 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:53:55.186 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:53:55.186957 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:53:55.236 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:53:55.237402 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:54:05.312 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:54:05.312758 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:54:05.323 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:54:05.324508 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:54:15.460 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:54:15.461364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:54:15.475 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:54:15.476302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:54:25.564 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:54:25.564611 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:54:25.578 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:54:25.578951 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:54:35.703 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:54:35.703630 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:54:35.713 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:54:35.714147 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:54:45.823 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:54:45.824513 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:54:45.838 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:54:45.839546 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:54:55.913 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:54:55.914180 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:54:55.926 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:54:55.926791 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:55:06.048 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:55:06.049327 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:55:06.064 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:55:06.065027 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:55:16.184 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:55:16.185511 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:55:16.198 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:55:16.199177 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:55:26.266 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:55:26.266642 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:55:26.281 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:55:26.282740 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:55:36.397 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:55:36.397657 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:55:36.409 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:55:36.410068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:55:46.474 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:55:46.475203 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:55:46.488 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:55:46.489029 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:55:56.605 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:55:56.606084 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:55:56.617 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:55:56.618529 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:56:06.721 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:56:06.722019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:56:06.735 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:56:06.736110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:56:16.819 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:56:16.819666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:56:16.830 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:56:16.831163 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:56:26.900 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:56:26.899901 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:56:26.911 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:56:26.912547 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:56:37.013 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:56:37.013510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:56:37.025 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:56:37.025924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:56:47.088 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:56:47.088915 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:56:47.102 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:56:47.103128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:56:57.235 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:56:57.235436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:56:57.246 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:56:57.247014 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:57:07.326 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:57:07.327082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:57:07.449 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:57:07.450076 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:57:17.593 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:57:17.593926 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:57:17.603 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:57:17.604275 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:57:27.682 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:57:27.682926 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:57:27.755 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:57:27.756057 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:57:37.871 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:57:37.868289 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:57:37.882 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:57:37.883200 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:57:47.989 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:57:47.990163 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:57:48.004 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:57:48.004525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:57:58.173 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:57:58.173814 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:57:58.188 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:57:58.189022 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:58:08.269 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:58:08.270323 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:58:08.425 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:58:08.426481 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:58:18.581 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:58:18.582346 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:58:18.596 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:58:18.597716 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:58:28.716 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:58:28.716941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:58:28.730 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:58:28.731717 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:58:38.849 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:58:38.849707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:58:38.861 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:58:38.862665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:58:48.935 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:58:48.935845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:58:48.947 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:58:48.947916 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:58:59.053 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:58:59.054298 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:58:59.067 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:58:59.068208 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:59:09.127 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:59:09.124872 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:59:09.182 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:59:09.183020 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:59:19.327 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:59:19.328431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:59:19.342 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:59:19.342990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:59:29.430 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:59:29.430654 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:59:29.441 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:59:29.442103 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:59:39.578 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:59:39.579297 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:59:39.591 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:59:39.591981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:59:49.678 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:59:49.678178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:59:49.690 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:59:49.691108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:59:59.758 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:59:59.758851 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:59:59.771 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 17:59:59.772233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:00:09.861 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:00:09.862425 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:00:09.874 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:00:09.875173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:00:19.969 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:00:19.969590 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:00:19.982 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:00:19.983655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:00:30.050 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:00:30.051143 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:00:30.127 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:00:30.127686 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:00:40.222 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:00:40.222848 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:00:40.237 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:00:40.238334 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:00:50.311 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:00:50.311754 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:00:50.324 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:00:50.325043 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:01:00.422 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:01:00.423123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:01:00.655 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:01:00.498861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:01:10.575 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:01:10.575703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:01:10.590 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:01:10.590900 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:01:20.728 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:01:20.729511 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:01:20.743 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:01:20.744418 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:01:30.854 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:01:30.854993 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:01:30.868 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:01:30.869491 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:01:40.941 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:01:40.942667 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:01:40.957 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:01:40.958334 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:01:51.067 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:01:51.068476 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:01:51.079 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:01:51.080139 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:02:01.147 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:02:01.148104 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:02:01.159 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:02:01.160514 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:02:11.261 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:02:11.261926 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:02:11.274 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:02:11.275274 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:02:21.422 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:02:21.422522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:02:21.433 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:02:21.434911 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:02:31.606 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:02:31.607180 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:02:31.619 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:02:31.620552 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:02:41.716 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:02:41.716993 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:02:41.745 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:02:41.746052 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:02:51.881 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:02:51.881949 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:02:51.893 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:02:51.894695 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:03:01.971 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:03:01.971246 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:03:01.983 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:03:01.984228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:03:12.048 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:03:12.049435 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:03:12.064 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:03:12.064834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:03:22.134 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:03:22.134964 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:03:22.188 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:03:22.189158 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:03:32.269 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:03:32.270586 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:03:32.283 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:03:32.284162 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:03:42.387 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:03:42.387432 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:03:42.400 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:03:42.401798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:03:52.535 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:03:52.535791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:03:52.548 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:03:52.549680 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:04:02.649 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:04:02.649703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:04:02.661 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:04:02.662988 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:04:12.759 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:04:12.760417 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:04:12.779 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:04:12.776537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:04:22.875 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:04:22.876519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:04:22.886 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:04:22.887742 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:04:32.999 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:04:33.000241 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:04:33.024 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:04:33.024945 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:04:43.154 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:04:43.154870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:04:43.164 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:04:43.166082 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:04:53.236 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:04:53.237048 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:04:53.249 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:04:53.250949 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:05:03.383 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:05:03.383869 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:05:03.395 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:05:03.396784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:05:13.486 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:05:13.487231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:05:13.499 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:05:13.500122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:05:23.593 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:05:23.594437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:05:23.666 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:05:23.666981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:05:33.829 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:05:33.830111 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:05:33.840 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:05:33.841370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:05:43.914 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:05:43.915033 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:05:43.925 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:05:43.926848 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:05:54.023 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:05:54.023954 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:05:54.036 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:05:54.037087 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:06:04.164 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:06:04.164911 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:06:04.177 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:06:04.178745 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:06:14.293 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:06:14.294238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:06:14.309 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:06:14.309957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:06:24.425 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:06:24.426343 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:06:24.441 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:06:24.442668 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:06:34.545 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:06:34.545742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:06:34.555 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:06:34.556162 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:06:44.661 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:06:44.661903 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:06:44.673 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:06:44.674415 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:06:54.746 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:06:54.746769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:06:55.039 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:06:54.760528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:07:04.855 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:07:04.855934 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:07:04.867 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:07:04.868600 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:07:14.973 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:07:14.973630 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:07:14.987 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:07:14.988463 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:07:25.077 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:07:25.070319 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:07:25.101 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:07:25.101861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:07:35.175 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:07:35.176269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:07:35.187 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:07:35.188924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:07:45.345 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:07:45.346259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:07:45.359 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:07:45.360408 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:07:55.473 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:07:55.473882 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:07:55.485 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:07:55.486886 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:08:05.560 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:08:05.561593 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:08:05.573 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:08:05.574377 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:08:15.634 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:08:15.635526 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:08:15.647 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:08:15.648688 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:08:25.768 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:08:25.769082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:08:25.782 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:08:25.783451 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:08:35.882 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:08:35.882678 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:08:35.894 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:08:35.894942 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:08:46.009 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:08:46.009992 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:08:46.021 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:08:46.022498 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:08:56.131 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:08:56.132237 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:08:56.145 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:08:56.145985 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:09:06.277 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:09:06.277699 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:09:06.289 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:09:06.290138 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:09:16.427 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:09:16.428012 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:09:16.440 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:09:16.441428 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:09:26.561 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:09:26.562354 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:09:26.576 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:09:26.577185 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:09:36.732 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:09:36.732711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:09:36.746 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:09:36.747979 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:09:42.924 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:09:42.925239 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T18:09:43.612 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:09:43.613039 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T18:09:43.604+0000 7f986790c380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T18:09:43.613 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T18:09:43.604+0000 7f986790c380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T18:09:43.613 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T18:09:43.604+0000 7f986790c380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T18:09:43.613 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T18:09:46.927 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:09:46.927836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:09:46.939 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:09:46.940594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:09:57.038 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:09:57.039055 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:09:57.051 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:09:57.052675 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:10:07.156 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:10:07.155953 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:10:07.168 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:10:07.169547 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:10:17.248 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:10:17.248791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:10:17.301 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:10:17.302615 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:10:27.465 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:10:27.465703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:10:27.495 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:10:27.496439 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:10:37.613 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:10:37.614200 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:10:37.624 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:10:37.625533 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:10:47.725 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:10:47.726064 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:10:47.738 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:10:47.739537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:10:57.811 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:10:57.811470 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:10:57.826 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:10:57.826727 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:11:07.929 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:11:07.929612 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:11:07.939 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:11:07.940565 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:11:18.018 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:11:18.019367 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:11:18.098 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:11:18.099042 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:11:28.226 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:11:28.226614 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:11:28.239 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:11:28.240544 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:11:38.395 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:11:38.395886 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:11:38.409 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:11:38.410519 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:11:48.547 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:11:48.547926 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:11:48.558 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:11:48.559173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:11:58.628 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:11:58.629219 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:11:58.641 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:11:58.642511 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:12:08.769 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:12:08.766419 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:12:08.779 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:12:08.779697 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:12:18.908 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:12:18.909044 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:12:18.919 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:12:18.919915 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:12:28.994 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:12:28.995076 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:12:29.007 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:12:29.007720 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:12:39.098 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:12:39.099089 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:12:39.113 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:12:39.114108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:12:49.238 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:12:49.238821 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:12:49.257 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:12:49.254327 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:12:59.331 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:12:59.331663 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:12:59.344 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:12:59.345056 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:13:09.483 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:13:09.483309 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:13:09.494 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:13:09.495104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:13:19.607 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:13:19.607870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:13:19.620 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:13:19.621441 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:13:29.724 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:13:29.723664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:13:29.737 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:13:29.737701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:13:39.810 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:13:39.809761 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:13:39.857 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:13:39.857867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:13:50.002 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:13:50.002720 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:13:50.013 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:13:50.014314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:14:00.095 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:14:00.096301 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:14:00.103 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:14:00.104283 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:14:10.178 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:14:10.178800 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:14:10.189 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:14:10.189931 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:14:20.368 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:14:20.368878 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:14:20.439 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:14:20.439783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:14:30.547 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:14:30.547629 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:14:30.560 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:14:30.560809 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:14:40.803 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:14:40.803182 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:14:40.811 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:14:40.811696 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:14:50.925 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:14:50.925947 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:14:50.940 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:14:50.940722 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:15:01.079 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:15:01.079228 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:15:01.088 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:15:01.088504 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:15:11.167 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:15:11.167082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:15:11.178 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:15:11.178596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:15:21.245 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:15:21.245970 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:15:21.259 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:15:21.259809 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:15:31.368 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:15:31.368424 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:15:31.388 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:15:31.388312 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:15:41.500 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:15:41.501104 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:15:41.511 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:15:41.512166 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:15:51.625 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:15:51.625213 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:15:51.635 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:15:51.636347 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:16:01.732 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:16:01.732504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:16:01.745 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:16:01.745501 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:16:11.825 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:16:11.825380 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:16:11.838 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:16:11.839059 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:16:21.944 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:16:21.944033 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:16:21.957 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:16:21.957619 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:16:32.061 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:16:32.061081 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:16:32.074 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:16:32.074756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:16:42.190 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:16:42.190227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:16:42.202 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:16:42.202712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:16:52.331 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:16:52.331709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:16:52.344 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:16:52.344681 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:17:02.474 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:17:02.473797 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:17:02.485 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:17:02.486048 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:17:12.606 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:17:12.606691 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:17:12.620 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:17:12.621112 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:17:22.734 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:17:22.734052 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:17:22.746 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:17:22.746712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:17:32.852 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:17:32.852443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:17:32.866 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:17:32.866791 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:17:42.984 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:17:42.984188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:17:42.996 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:17:42.996574 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:17:53.060 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:17:53.060050 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:17:53.122 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:17:53.122258 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:18:03.193 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:18:03.193139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:18:03.204 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:18:03.204284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:18:13.302 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:18:13.301678 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:18:13.316 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:18:13.316612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:18:23.396 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:18:23.396810 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:18:23.408 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:18:23.408833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:18:33.528 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:18:33.528316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:18:33.546 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:18:33.546910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:18:43.660 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:18:43.659846 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:18:43.675 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:18:43.675408 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:18:53.744 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:18:53.744651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:18:53.759 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:18:53.759851 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:19:03.874 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:19:03.874587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:19:03.887 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:19:03.888228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:19:13.956 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:19:13.956037 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:19:13.988 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:19:13.988111 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:19:24.108 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:19:24.108774 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:19:24.121 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:19:24.121747 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:19:34.281 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:19:34.281337 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:19:34.291 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:19:34.291362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:19:44.398 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:19:44.396266 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:19:44.408 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:19:44.408486 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:19:54.522 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:19:54.522040 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:19:54.534 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:19:54.535114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:20:04.740 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:20:04.693486 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:20:04.741 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:20:04.707134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:20:14.865 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:20:14.865250 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:20:14.883 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:20:14.883733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:20:24.963 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:20:24.963234 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:20:24.975 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:20:24.975126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:20:35.274 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:20:35.273137 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:20:35.287 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:20:35.287975 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:20:45.486 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:20:45.486026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:20:45.496 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:20:45.496915 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:20:55.619 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:20:55.618838 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:20:55.631 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:20:55.631731 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:21:05.725 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:21:05.724723 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:21:05.738 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:21:05.738001 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:21:15.816 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:21:15.815397 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:21:15.829 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:21:15.829109 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:21:25.931 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:21:25.931266 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:21:25.944 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:21:25.944965 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:21:36.043 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:21:36.043353 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:21:36.053 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:21:36.053263 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:21:46.246 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:21:46.245572 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:21:46.258 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:21:46.257980 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:21:56.381 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:21:56.381323 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:21:56.397 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:21:56.397759 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:22:06.464 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:22:06.464315 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:22:06.482 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:22:06.482293 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:22:16.589 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:22:16.588988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:22:16.603 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:22:16.603240 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:22:26.719 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:22:26.718879 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:22:26.733 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:22:26.733420 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:22:36.805 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:22:36.804933 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:22:36.819 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:22:36.818759 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:22:46.900 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:22:46.899860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:22:46.959 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:22:46.959607 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:22:57.090 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:22:57.090159 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:22:57.102 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:22:57.102786 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:23:07.163 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:23:07.163648 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:23:07.178 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:23:07.178006 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:23:17.328 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:23:17.327629 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:23:17.340 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:23:17.341010 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:23:27.479 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:23:27.479407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:23:27.492 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:23:27.492484 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:23:37.582 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:23:37.581542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:23:37.592 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:23:37.592445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:23:47.667 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:23:47.667037 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:23:47.680 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:23:47.680780 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:23:57.756 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:23:57.755676 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:23:57.767 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:23:57.766923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:24:07.880 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:24:07.879008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:24:07.895 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:24:07.894797 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:24:18.025 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:24:18.024297 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:24:18.041 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:24:18.040921 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:24:28.133 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:24:28.133062 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:24:28.147 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:24:28.147183 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:24:38.255 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:24:38.255274 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:24:38.267 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:24:38.266695 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:24:48.371 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:24:48.370851 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:24:48.384 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:24:48.384479 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:24:58.474 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:24:58.473689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:24:58.488 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:24:58.488755 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:25:08.581 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:25:08.581272 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:25:08.592 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:25:08.592412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:25:18.684 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:25:18.683481 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:25:18.697 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:25:18.697427 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:25:28.825 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:25:28.824464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:25:28.839 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:25:28.839499 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:25:38.944 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:25:38.944253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:25:38.958 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:25:38.958076 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:25:49.070 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:25:49.070413 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:25:49.084 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:25:49.084166 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:25:59.201 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:25:59.200667 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:25:59.214 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:25:59.213748 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:26:09.296 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:26:09.295830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:26:09.307 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:26:09.306776 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:26:19.501 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:26:19.500854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:26:19.513 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:26:19.513337 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:26:23.627 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:26:23.626934 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T18:26:24.363 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:26:24.362532 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T18:26:24.352+0000 7fea1b588380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T18:26:24.364 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T18:26:24.352+0000 7fea1b588380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T18:26:24.364 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T18:26:24.352+0000 7fea1b588380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T18:26:24.364 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T18:26:29.619 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:26:29.618820 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:26:29.632 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:26:29.632039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:26:39.746 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:26:39.746150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:26:39.762 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:26:39.762187 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:26:49.872 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:26:49.871552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:26:49.887 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:26:49.887287 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:26:59.956 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:26:59.955840 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:26:59.970 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:26:59.969627 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:27:10.080 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:27:10.079773 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:27:10.093 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:27:10.093248 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:27:20.195 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:27:20.195083 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:27:20.207 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:27:20.207446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:27:30.309 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:27:30.308424 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:27:30.322 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:27:30.321986 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:27:40.393 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:27:40.391730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:27:40.447 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:27:40.446902 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:27:50.564 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:27:50.564165 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:27:50.578 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:27:50.578423 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:28:00.700 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:28:00.699965 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:28:00.715 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:28:00.714959 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:28:10.781 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:28:10.781231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:28:10.794 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:28:10.793906 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:28:20.910 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:28:20.910090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:28:20.923 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:28:20.922778 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:28:31.012 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:28:31.012082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:28:31.025 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:28:31.024988 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:28:41.096 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:28:41.095549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:28:41.158 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:28:41.158315 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:28:51.228 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:28:51.227445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:28:51.237 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:28:51.237381 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:29:01.369 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:29:01.368386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:29:01.380 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:29:01.380440 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:29:11.481 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:29:11.480139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:29:11.494 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:29:11.493793 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:29:21.591 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:29:21.591047 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:29:21.619 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:29:21.608500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:29:31.683 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:29:31.682600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:29:31.733 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:29:31.731258 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:29:41.815 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:29:41.815006 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:29:41.871 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:29:41.871084 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:29:51.993 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:29:51.992366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:29:52.008 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:29:52.007236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:30:02.190 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:30:02.190145 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:30:02.251 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:30:02.250568 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:30:12.364 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:30:12.363885 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:30:12.394 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:30:12.394383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:30:22.471 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:30:22.470244 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:30:22.481 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:30:22.481251 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:30:32.608 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:30:32.607103 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:30:32.622 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:30:32.621669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:30:42.743 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:30:42.742648 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:30:42.758 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:30:42.757446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:30:52.866 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:30:52.864877 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:30:52.926 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:30:52.925456 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:31:03.002 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:31:03.001425 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:31:03.015 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:31:03.014706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:31:13.429 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:31:13.428080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:31:13.440 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:31:13.440240 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:31:23.582 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:31:23.580820 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:31:23.593 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:31:23.592799 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:31:33.708 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:31:33.707952 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:31:33.720 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:31:33.720327 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:31:43.850 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:31:43.849091 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:31:43.860 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:31:43.859929 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:31:53.970 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:31:53.969372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:31:53.982 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:31:53.981971 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:32:04.136 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:32:04.135203 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:32:04.148 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:32:04.147376 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:32:14.281 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:32:14.280214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:32:14.299 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:32:14.294165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:32:24.372 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:32:24.371649 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:32:24.390 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:32:24.388600 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:32:34.468 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:32:34.467824 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:32:34.538 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:32:34.537433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:32:44.655 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:32:44.654702 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:32:44.668 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:32:44.668110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:32:54.743 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:32:54.742186 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:32:54.756 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:32:54.755708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:33:04.871 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:33:04.870861 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:33:04.884 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:33:04.884152 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:33:14.955 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:33:14.954293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:33:14.969 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:33:14.968049 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:33:25.117 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:33:25.116810 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:33:25.130 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:33:25.129492 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:33:35.233 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:33:35.232033 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:33:35.248 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:33:35.247733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:33:45.327 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:33:45.326456 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:33:45.340 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:33:45.339925 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:33:55.447 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:33:55.446542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:33:55.460 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:33:55.459966 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:34:05.545 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:34:05.544463 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:34:05.559 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:34:05.558940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:34:15.720 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:34:15.719389 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:34:15.777 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:34:15.776733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:34:25.911 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:34:25.910887 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:34:25.926 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:34:25.925330 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:34:36.054 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:34:36.053201 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:34:36.071 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:34:36.066281 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:34:46.180 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:34:46.178313 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:34:46.195 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:34:46.194816 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:34:56.278 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:34:56.277504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:34:56.291 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:34:56.290483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:35:06.366 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:35:06.365700 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:35:06.380 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:35:06.379835 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:35:16.510 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:35:16.509402 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:35:16.526 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:35:16.525654 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:35:26.612 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:35:26.611311 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:35:26.623 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:35:26.622977 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:35:36.699 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:35:36.698200 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:35:36.710 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:35:36.710020 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:35:46.779 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:35:46.778151 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:35:46.792 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:35:46.792167 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:35:56.916 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:35:56.915280 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:35:56.930 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:35:56.930119 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:36:07.052 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:36:07.046601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:36:07.060 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:36:07.059838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:36:17.141 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:36:17.140775 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:36:17.154 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:36:17.153451 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:36:27.229 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:36:27.228397 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:36:27.245 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:36:27.244653 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:36:37.320 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:36:37.319234 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:36:37.376 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:36:37.375184 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:36:47.489 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:36:47.488198 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:36:47.507 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:36:47.506687 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:36:57.582 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:36:57.581766 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:36:57.596 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:36:57.595449 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:37:07.671 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:37:07.670436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:37:07.684 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:37:07.684098 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:37:17.789 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:37:17.787755 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:37:17.801 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:37:17.800357 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:37:27.868 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:37:27.867185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:37:27.882 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:37:27.881710 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:37:37.951 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:37:37.950347 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:37:37.965 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:37:37.964091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:37:48.144 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:37:48.143576 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:37:48.158 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:37:48.157653 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:37:58.293 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:37:58.291934 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:37:58.305 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:37:58.304305 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:38:08.399 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:38:08.398284 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:38:08.464 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:38:08.463653 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:38:18.693 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:38:18.691062 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:38:18.705 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:38:18.704340 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:38:28.780 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:38:28.778903 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:38:28.790 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:38:28.789759 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:38:38.859 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:38:38.858110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:38:38.872 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:38:38.871863 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:38:49.013 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:38:49.011705 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:38:49.024 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:38:49.023706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:38:59.122 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:38:59.121387 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:38:59.137 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:38:59.136841 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:39:09.229 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:39:09.227575 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:39:09.242 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:39:09.240449 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:39:19.360 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:39:19.359437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:39:19.376 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:39:19.375334 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:39:29.473 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:39:29.472239 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:39:29.486 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:39:29.485695 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:39:39.553 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:39:39.552083 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:39:39.566 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:39:39.565324 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:39:49.692 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:39:49.690997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:39:49.702 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:39:49.701325 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:39:59.763 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:39:59.761921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:39:59.774 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:39:59.773305 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:40:09.881 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:40:09.879862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:40:09.895 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:40:09.895001 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:40:19.990 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:40:19.988945 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:40:20.003 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:40:20.002553 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:40:30.070 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:40:30.069461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:40:30.148 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:40:30.147560 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:40:40.261 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:40:40.259875 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:40:40.273 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:40:40.272646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:40:50.350 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:40:50.349478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:40:50.434 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:40:50.432736 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:41:00.526 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:41:00.522168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:41:00.534 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:41:00.533128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:41:10.614 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:41:10.613191 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:41:10.627 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:41:10.626890 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:41:20.799 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:41:20.797944 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:41:20.810 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:41:20.809169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:41:30.988 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:41:30.987026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:41:31.000 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:41:30.999864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:41:41.127 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:41:41.126104 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:41:41.140 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:41:41.139018 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:41:51.214 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:41:51.212015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:41:51.263 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:41:51.261933 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:42:01.371 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:42:01.370174 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:42:01.385 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:42:01.384461 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:42:11.468 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:42:11.467586 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:42:11.485 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:42:11.484770 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:42:21.597 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:42:21.596169 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:42:21.611 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:42:21.610878 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:42:31.680 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:42:31.678806 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:42:31.734 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:42:31.732859 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:42:41.799 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:42:41.797618 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:42:41.811 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:42:41.810225 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:42:51.914 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:42:51.913336 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:42:51.930 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:42:51.928591 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:43:02.009 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:43:02.007722 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:43:02.026 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:43:02.023176 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:43:04.371 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:43:04.370380 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T18:43:05.095 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:43:05.093491 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T18:43:05.084+0000 7f0624395380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T18:43:05.095 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T18:43:05.084+0000 7f0624395380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T18:43:05.095 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T18:43:05.084+0000 7f0624395380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T18:43:05.096 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T18:43:12.213 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:43:12.212306 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:43:12.229 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:43:12.228189 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:43:22.335 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:43:22.334330 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:43:22.347 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:43:22.346008 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:43:32.446 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:43:32.445332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:43:32.460 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:43:32.459210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:43:42.566 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:43:42.565273 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:43:42.579 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:43:42.578918 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:43:52.681 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:43:52.679940 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:43:52.746 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:43:52.744865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:44:02.852 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:44:02.851451 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:44:02.881 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:44:02.880419 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:44:12.952 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:44:12.951138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:44:12.964 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:44:12.963784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:44:23.033 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:44:23.032540 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:44:23.046 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:44:23.045016 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:44:33.162 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:44:33.161327 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:44:33.175 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:44:33.173954 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:44:43.290 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:44:43.289464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:44:43.302 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:44:43.301237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:44:53.402 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:44:53.400695 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:44:53.415 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:44:53.414230 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:45:03.505 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:45:03.504313 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:45:03.516 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:45:03.515151 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:45:13.621 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:45:13.619819 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:45:13.634 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:45:13.633881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:45:23.749 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:45:23.747858 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:45:23.763 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:45:23.762164 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:45:33.834 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:45:33.832909 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:45:33.845 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:45:33.844408 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:45:43.973 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:45:43.972089 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:45:43.984 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:45:43.983491 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:45:54.061 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:45:54.060368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:45:54.076 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:45:54.074992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:46:04.242 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:46:04.241102 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:46:04.255 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:46:04.254147 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:46:14.367 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:46:14.366430 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:46:14.380 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:46:14.379698 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:46:24.487 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:46:24.485691 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:46:24.503 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:46:24.502047 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:46:34.565 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:46:34.564318 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:46:34.639 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:46:34.638360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:46:44.723 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:46:44.721600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:46:44.737 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:46:44.736302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:46:54.808 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:46:54.807022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:46:54.822 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:46:54.821633 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:47:04.926 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:47:04.924951 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:47:04.942 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:47:04.941805 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:47:15.056 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:47:15.055090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:47:15.069 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:47:15.068879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:47:25.196 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:47:25.195568 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:47:25.209 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:47:25.208492 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:47:35.305 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:47:35.303643 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:47:35.317 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:47:35.316176 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:47:45.428 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:47:45.426453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:47:45.439 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:47:45.437867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:47:55.507 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:47:55.506267 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:47:55.523 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:47:55.522054 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:48:05.633 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:48:05.631833 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:48:05.645 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:48:05.644348 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:48:15.765 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:48:15.764443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:48:15.778 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:48:15.777148 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:48:25.935 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:48:25.933545 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:48:25.945 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:48:25.944015 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:48:36.062 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:48:36.061244 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:48:36.075 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:48:36.074703 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:48:46.151 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:48:46.150168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:48:46.163 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:48:46.162445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:48:56.274 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:48:56.272424 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:48:56.288 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:48:56.287111 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:49:06.363 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:49:06.361711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:49:06.376 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:49:06.375084 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:49:16.458 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:49:16.455646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:49:16.472 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:49:16.470094 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:49:26.543 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:49:26.540578 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:49:26.599 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:49:26.597858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:49:36.671 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:49:36.669780 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:49:36.684 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:49:36.683731 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:49:46.753 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:49:46.752109 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:49:46.764 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:49:46.763183 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:49:56.856 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:49:56.854842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:49:56.866 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:49:56.864741 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:50:07.019 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:50:07.017247 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:50:07.032 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:50:07.031266 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:50:17.264 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:50:17.262096 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:50:17.330 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:50:17.328390 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:50:27.557 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:50:27.556183 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:50:27.570 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:50:27.569053 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:50:37.833 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:50:37.832232 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:50:37.847 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:50:37.846542 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:50:47.922 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:50:47.921162 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:50:47.935 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:50:47.933886 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:50:58.005 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:50:58.004407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:50:58.020 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:50:58.016469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:51:08.128 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:51:08.127223 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:51:08.141 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:51:08.140081 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:51:18.213 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:51:18.212524 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:51:18.227 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:51:18.226168 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:51:28.298 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:51:28.296880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:51:28.312 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:51:28.311263 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:51:38.435 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:51:38.433612 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:51:38.449 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:51:38.448320 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:51:48.566 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:51:48.564778 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:51:48.576 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:51:48.575575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:51:58.723 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:51:58.721959 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:51:58.737 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:51:58.735230 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:52:08.858 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:52:08.856980 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:52:08.877 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:52:08.876330 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:52:19.050 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:52:19.048867 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:52:19.074 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:52:19.073140 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:52:29.166 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:52:29.164608 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:52:29.180 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:52:29.179001 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:52:39.252 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:52:39.250787 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:52:39.269 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:52:39.263399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:52:49.381 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:52:49.380376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:52:49.396 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:52:49.394918 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:52:59.511 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:52:59.510102 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:52:59.524 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:52:59.523033 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:53:09.646 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:53:09.645311 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:53:09.656 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:53:09.655704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:53:19.780 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:53:19.779109 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:53:19.796 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:53:19.795344 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:53:29.874 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:53:29.873303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:53:29.889 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:53:29.887994 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:53:40.173 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:53:40.171752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:53:40.187 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:53:40.186605 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:53:50.323 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:53:50.322166 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:53:50.337 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:53:50.335511 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:54:00.413 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:54:00.412272 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:54:00.426 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:54:00.425488 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:54:10.529 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:54:10.527971 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:54:10.543 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:54:10.541939 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:54:20.659 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:54:20.657939 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:54:20.677 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:54:20.676103 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:54:30.804 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:54:30.803274 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:54:30.818 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:54:30.817092 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:54:40.923 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:54:40.921446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:54:40.942 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:54:40.935143 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:54:51.023 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:54:51.022355 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:54:51.035 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:54:51.034594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:55:01.165 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:55:01.164247 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:55:01.179 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:55:01.178187 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:55:11.321 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:55:11.319899 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:55:11.335 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:55:11.334171 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:55:21.435 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:55:21.434372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:55:21.464 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:55:21.456796 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:55:31.553 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:55:31.552219 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:55:31.570 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:55:31.569256 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:55:41.689 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:55:41.687998 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:55:41.701 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:55:41.699680 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:55:51.780 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:55:51.779297 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:55:51.795 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:55:51.793747 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:56:01.917 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:56:01.915199 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:56:01.929 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:56:01.928435 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:56:12.039 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:56:12.036692 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:56:12.051 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:56:12.050216 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:56:22.165 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:56:22.163537 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:56:22.176 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:56:22.175289 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:56:32.280 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:56:32.278443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:56:32.294 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:56:32.292712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:56:42.358 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:56:42.357078 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:56:42.371 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:56:42.370288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:56:52.438 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:56:52.436312 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:56:52.450 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:56:52.449497 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:57:02.550 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:57:02.549304 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:57:02.566 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:57:02.565063 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:57:12.650 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:57:12.649441 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:57:12.661 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:57:12.660153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:57:22.775 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:57:22.773911 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:57:22.788 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:57:22.787253 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:57:32.914 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:57:32.912453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:57:32.928 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:57:32.927109 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:57:43.007 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:57:43.006037 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:57:43.024 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:57:43.022837 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:57:53.130 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:57:53.129181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:57:53.145 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:57:53.144040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:58:03.222 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:58:03.220775 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:58:03.269 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:58:03.267865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:58:13.368 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:58:13.366459 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:58:13.383 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:58:13.381474 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:58:23.503 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:58:23.502255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:58:23.515 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:58:23.514572 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:58:33.623 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:58:33.621513 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:58:33.635 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:58:33.634502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:58:43.705 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:58:43.703616 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:58:43.718 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:58:43.717068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:58:53.789 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:58:53.788152 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:58:53.804 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:58:53.802966 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:59:03.902 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:59:03.900437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:59:03.914 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:59:03.912527 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:59:14.058 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:59:14.056431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:59:14.069 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:59:14.068458 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:59:24.127 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:59:24.122188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:59:24.197 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:59:24.195396 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:59:34.312 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:59:34.311005 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:59:34.327 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:59:34.325936 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:59:44.400 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:59:44.398767 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:59:44.410 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:59:44.409270 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:59:45.104 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:59:45.102805 I | op-mon: parsing mon endpoints: c=10.99.78.235:6789,a=10.107.211.141:6789,b=10.97.13.112:6789 2022-03-20T18:59:45.800 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:59:45.798068 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T18:59:45.788+0000 7f1036ce0380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T18:59:45.800 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T18:59:45.788+0000 7f1036ce0380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T18:59:45.801 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20T18:59:45.788+0000 7f1036ce0380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T18:59:45.801 INFO:tasks.rook.operator.smithi006.stdout:couldn't init storage provider. : exit status 5 2022-03-20T18:59:54.528 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:59:54.525617 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:59:54.540 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 18:59:54.539625 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:00:04.644 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:00:04.642398 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:00:04.661 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:00:04.659182 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:00:14.742 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:00:14.740976 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:00:14.815 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:00:14.813690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:00:24.886 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:00:24.884445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:00:24.899 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:00:24.897993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:00:35.022 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:00:35.020501 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:00:35.034 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:00:35.032988 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:00:45.186 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:00:45.184772 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:00:45.199 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:00:45.197963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:00:55.307 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:00:55.305874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:00:55.319 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:00:55.317441 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:01:05.411 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:01:05.409258 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:01:05.426 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:01:05.424583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:01:15.534 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:01:15.532854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:01:15.546 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:01:15.544924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:01:25.653 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:01:25.651494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:01:25.665 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:01:25.663144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:01:35.765 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:01:35.764221 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:01:35.778 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:01:35.776601 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:01:45.889 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:01:45.887145 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:01:45.902 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:01:45.900697 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:01:56.044 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:01:56.042815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:01:56.057 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:01:56.056058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:02:06.184 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:02:06.182166 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:02:06.202 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:02:06.201197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:02:16.308 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:02:16.306889 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:02:16.328 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:02:16.327325 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:02:26.416 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:02:26.414283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:02:26.431 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:02:26.429675 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:02:36.580 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:02:36.579063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:02:36.594 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:02:36.592857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:02:46.679 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:02:46.678010 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:02:46.691 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:02:46.690265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:02:56.814 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:02:56.812713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:02:56.826 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:02:56.824639 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:03:06.958 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:03:06.956441 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:03:06.971 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:03:06.969935 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:03:17.038 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:03:17.036364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:03:17.051 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:03:17.050384 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:03:27.156 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:03:27.154409 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:03:27.174 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:03:27.169594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:03:37.273 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:03:37.270979 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:03:37.285 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:03:37.284158 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:03:47.348 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:03:47.346991 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:03:47.360 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:03:47.359273 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:03:57.471 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:03:57.469812 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:03:57.485 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:03:57.483704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:04:07.546 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:04:07.541167 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:04:07.557 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:04:07.553965 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:04:17.633 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:04:17.632085 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:04:17.684 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:04:17.682610 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:04:27.800 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:04:27.798399 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:04:27.815 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:04:27.813438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:04:37.880 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:04:37.878458 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:04:37.892 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:04:37.890592 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:04:48.126 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:04:48.124654 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:04:48.136 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:04:48.135055 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:04:58.238 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:04:58.236911 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:04:58.251 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:04:58.249494 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:05:08.555 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:05:08.553962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:05:08.569 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:05:08.568346 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:05:18.690 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:05:18.688361 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:05:18.702 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:05:18.701163 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:05:28.837 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:05:28.835983 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:05:28.854 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:05:28.852596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:05:38.966 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:05:38.964879 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:05:38.980 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:05:38.978569 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:05:49.049 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:05:49.048142 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:05:49.063 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:05:49.061945 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:05:59.176 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:05:59.174962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:05:59.190 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:05:59.188604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:06:09.259 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:06:09.258125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:06:09.271 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:06:09.270314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:06:19.373 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:06:19.371606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:06:19.385 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:06:19.383993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:06:29.491 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:06:29.490061 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:06:29.506 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:06:29.505032 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:06:39.622 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:06:39.620701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:06:39.639 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:06:39.633783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:06:49.701 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:06:49.698859 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T19:06:49.711 INFO:tasks.rook.operator.smithi006.stdout:2022-03-20 19:06:49.710402 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T21:27:08.527 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-03-20T21:27:08.552 DEBUG:teuthology.task.console_log:Killing console logger for smithi006 2022-03-20T21:27:08.554 DEBUG:teuthology.task.console_log:Killing console logger for smithi045 2022-03-20T21:27:08.555 DEBUG:teuthology.task.console_log:Killing console logger for smithi123 2022-03-20T21:27:08.555 DEBUG:teuthology.exit:Finished running handlers