2022-03-20T14:43:23.278 INFO:root:teuthology version: 1.1.0-a4e18662 2022-03-20T14:43:23.281 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/flannel rook/master} --archive /home/teuthworker/archive/yuriw-2022-03-20_14:28:38-rados-wip-yuri6-testing-2022-03-18-1104-distro-default-smithi/6748893 --verbose --owner scheduled_yuriw@teuthology --name yuriw-2022-03-20_14:28:38-rados-wip-yuri6-testing-2022-03-18-1104-distro-default-smithi -- /home/teuthworker/archive/yuriw-2022-03-20_14:28:38-rados-wip-yuri6-testing-2022-03-18-1104-distro-default-smithi/6748893/orig.config.yaml 2022-03-20T14:43:23.330 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-03-20T14:43:23.392 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2022-03-20_14:28:38-rados-wip-yuri6-testing-2022-03-18-1104-distro-default-smithi/6748893 branch: wip-yuri6-testing-2022-03-18-1104 description: rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/flannel rook/master} email: null first_in_suite: false job_id: '6748893' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2022-03-20_14:28:38-rados-wip-yuri6-testing-2022-03-18-1104-distro-default-smithi nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri6-testing-2022-03-18-1104 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: e92561bf8a1bc96ca490936c6f46ac4673e1b26e ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: e92561bf8a1bc96ca490936c6f46ac4673e1b26e kernel: hwe: true kubeadm: pod_network: flannel version: '1.21' rook: rook_image: rook/ceph:master workunit: branch: wip-yuri6-testing-2022-03-18-1104 sha1: e92561bf8a1bc96ca490936c6f46ac4673e1b26e owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.a - - host.b - client.b - - host.c - client.c sha1: e92561bf8a1bc96ca490936c6f46ac4673e1b26e sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri6-testing-2022-03-18-1104 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: e92561bf8a1bc96ca490936c6f46ac4673e1b26e targets: smithi007.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWFCh0TrefiSuMQmeR8DbpuL3VJec45fhMK7fvaUeRP95Iz+WJLqQJ5gbloYeCiYR1RCSYuHWTqVpyX/WYKUeb+Kx2F2Jb+xjBo86CzZyLx+YegHE1Fw+aD7bQ7EFfHzw4dMglAqFlM47isUGXTH/wSET+Gsyx+yqNGdo54RojIqGxfRUzSSzHYD/7cRUuB8V1GOZxgHyOa1V+VUrJ7YbZBUipRmBY9BJlVuOSwc9kW3VQhZxCYsGWbGdJbSefooew/v28Czx4ZAWxEr0SvFviQkKws5oMHw1ccw5VSbn1nG81Xd4rBVM6x1tD493Llhp/yuwMMjj8CkGityKQam2XO4mh0sif3jATWgtsZJ0HtQkDb8sv/09akWTN/ycXyeX0vP5w9ng8C3FGgyRY6oVY2/K46cMQW7AZGaI/8gbn/tBJmez0PKflJ1DcBJP5m+xMSbp5IOjUUolySHT8qdxb2ZknKLST7vuf/apDmD1oxTKG8vtd4qdiCheKdNrHbc8= smithi057.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrjnWVzuJEVMrRqvgx0Ebh8j0Gzyv7/ClzEDV5+OIj51JVJphqCx9GVH5KbLVYm0E0DqUqILFD7RsLx5IEoWQXL443r602VbGXzhYmGFHjkmUkScYncxQaESlhOGHwD4txGY1qpHVu3OT8LSRflL3QIN7BI4vYZG6UCjm6LTxfqFyEGwrC0uhT7rt/8KtPTgBnomIW+wamnsAUEA5pbvoBDrulSbvBgFYoeTY1P41P4TBdeqyo37J6l9f8GsHX67dxE4eWazfswigHC+fMGxJlRhJxbbBGObYvn8atyLhUTmn99MebpFUCcCwXq7M35BpM6Pq3bSfXYPC1iPABJK6HyBtvZpYnb6UvLoaLpTp7V1y3Rd1wn4UgMSD31Pb+jBh/XawamI/y7oRDAiwZrBvoe6vPfkX66W3gUfG5tX4AchelxxjYFBtyaY+T/doEDEfXcJVfbkE63xEKpQ+OkkjXhFc5hDpi2ZfFv/EAOhL6EIwtzBDSiAX1bifSJzjbom8= smithi087.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+Qd9eo0ojUytZ0YqYd/y6KDIjst9uVMje+HVOXTWHGbh+kqIWXPIjEkgdhjZ1Q6gM+L5C2Do4MBsFBNdSZBUWtzK14v7H0KvdmK+8EsDXZwrQcG1AdEjbgKZeL1uEG2i/qBoZUStRSrek1XG2I1zj5nZEcjER/wSFgCulkQA0EZKNNtWa0gGoPnUy18GKUKQA08SzcapVcVfv/e5OXcpU6TEbL01f6eNpJaQmo1aNb5OSK1f2D1LDGOD7B99ZEAkKVVUmrO/XUUOLZOe7wtT23HV37nWXD0R3zrLB4iaCrBm8bo+Pqy0qPPBcfuUqxU9+FkIPIyKnbVy/pv1khVZsmhgAX6P/fKZhSwD6G679Z4b/Zu+Sg25eEzLaGH4PnTEGMy1CRVs/N9MvVMSmctbg+MS4iOZp3qfcrBs1tMepk9oBVue7/EF2kRGxONxkXxu7zfNGm23YRK6UJeqdCIZQwymKMIlNLjhn6rtyoFtlTUp+7xx7A4BxSHp0sDSwfdU= tasks: - kubeadm: null - nvme_loop: null - rook: null - rook.shell: - ceph -s - ceph orch status - ceph orch ps - ceph orch ls - ceph orch device ls - exec: host.a: - "set -ex\ntoolbox() {\n kubectl -n rook-ceph exec -it deploy/rook-ceph-tools\ \ -- \"$@\"\n}\norig_num_osd=`toolbox ceph osd stat | cut -f3 -d \" \"`\ntoolbox\ \ ceph orch osd rm 0 --force\nremoved_pv=\"\"\nwhile [ \"$removed_pv\" = \"\"\ \ ]\ndo\n removed_pv=`kubectl get pv | grep Released | cut -f1 -d \" \"`\n\ \ sleep 3s\ndone\ntarget_path=`kubectl get pv $removed_pv -o jsonpath='{.spec.local.path}'`\n\ host=`echo $removed_pv | cut -f1 -d \"-\"`\ntoolbox ceph orch device zap $host\ \ $target_path --force\nzap_completion=\"0\"\nwhile [ \"$zap_completion\" =\ \ \"0\" ]\ndo\n zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap\ \ -o jsonpath='{.status.succeeded.path}'`\n sleep 3s\ndone\nkubectl patch\ \ pv $removed_pv -p '{\"spec\":{\"claimRef\": null}}'\ntoolbox ceph orch apply\ \ osd --all-available-devices\nkubectl delete job rook-ceph-device-zap -n rook-ceph\n\ num_osd=\"0\"\nwhile [ \"$num_osd\" != \"$orig_num_osd\" ]\ndo\n echo \"\ waiting for osd to come back up\"\n num_osd=`toolbox ceph osd stat | cut\ \ -f3 -d \" \"`\n sleep 30s\ndone\n" - rook.shell: commands: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch apply rgw foo - ceph orch apply mds foo - ceph orch apply rbd-mirror - ceph orch apply nfs foo --port 12777 teuthology_branch: master teuthology_sha1: a4e18662e5de936eb0ac089e531b933ecbcadb0a timestamp: 2022-03-20_14:28:38 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.22301 2022-03-20T14:43:23.392 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa; will attempt to use it 2022-03-20T14:43:23.393 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks 2022-03-20T14:43:23.393 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-03-20T14:43:23.395 INFO:teuthology.task.internal:Checking packages... 2022-03-20T14:43:23.417 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'e92561bf8a1bc96ca490936c6f46ac4673e1b26e' 2022-03-20T14:43:23.417 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-03-20T14:43:23.418 INFO:teuthology.packaging:ref: None 2022-03-20T14:43:23.418 INFO:teuthology.packaging:tag: None 2022-03-20T14:43:23.418 INFO:teuthology.packaging:branch: wip-yuri6-testing-2022-03-18-1104 2022-03-20T14:43:23.419 INFO:teuthology.packaging:sha1: e92561bf8a1bc96ca490936c6f46ac4673e1b26e 2022-03-20T14:43:23.419 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri6-testing-2022-03-18-1104 2022-03-20T14:43:23.571 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-11187-ge92561bf-1focal 2022-03-20T14:43:23.572 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-03-20T14:43:23.583 INFO:teuthology.task.internal:no buildpackages task found 2022-03-20T14:43:23.584 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-03-20T14:43:23.633 INFO:teuthology.task.internal:Saving configuration 2022-03-20T14:43:23.646 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-03-20T14:43:23.658 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-20T14:43:23.680 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-03-20_14:28:38-rados-wip-yuri6-testing-2022-03-18-1104-distro-default-smithi/6748893', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-20 14:38:12.075602', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPTmVGITAfmgDqiuuSIveKdKRNQotL9B6RYrt5SXeLoZ'} 2022-03-20T14:43:23.702 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi057.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-03-20_14:28:38-rados-wip-yuri6-testing-2022-03-18-1104-distro-default-smithi/6748893', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-20 14:38:12.074190', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAcBqDsYbj/O5RhPN/ZwQf4gXYmlxjPiB7TYWEyR0Mnl'} 2022-03-20T14:43:23.724 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi087.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-03-20_14:28:38-rados-wip-yuri6-testing-2022-03-18-1104-distro-default-smithi/6748893', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-20 14:38:12.076669', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIH3vwkkYMJYSsWdHdVXAukYW4jJ7qWOTClqpldoZgJ2G'} 2022-03-20T14:43:23.724 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-03-20T14:43:23.736 INFO:teuthology.task.internal:roles: ubuntu@smithi007.front.sepia.ceph.com - ['host.a', 'client.a'] 2022-03-20T14:43:23.736 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['host.b', 'client.b'] 2022-03-20T14:43:23.737 INFO:teuthology.task.internal:roles: ubuntu@smithi087.front.sepia.ceph.com - ['host.c', 'client.c'] 2022-03-20T14:43:23.737 INFO:teuthology.run_tasks:Running task console_log... 2022-03-20T14:43:23.917 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f69e1514c80>, signals=[15]) 2022-03-20T14:43:23.918 INFO:teuthology.run_tasks:Running task internal.connect... 2022-03-20T14:43:24.622 INFO:teuthology.task.internal:Opening connections... 2022-03-20T14:43:24.622 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi007.front.sepia.ceph.com 2022-03-20T14:43:24.624 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-20T14:43:24.678 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi057.front.sepia.ceph.com 2022-03-20T14:43:24.679 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-20T14:43:24.731 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi087.front.sepia.ceph.com 2022-03-20T14:43:24.732 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-20T14:43:24.782 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-03-20T14:43:24.789 DEBUG:teuthology.orchestra.run.smithi007:> uname -m 2022-03-20T14:43:24.797 INFO:teuthology.orchestra.run.smithi007.stdout:x86_64 2022-03-20T14:43:24.798 DEBUG:teuthology.orchestra.run.smithi007:> cat /etc/os-release 2022-03-20T14:43:24.804 INFO:teuthology.orchestra.run.smithi007.stdout:NAME="Ubuntu" 2022-03-20T14:43:24.805 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-20T14:43:24.805 INFO:teuthology.orchestra.run.smithi007.stdout:ID=ubuntu 2022-03-20T14:43:24.805 INFO:teuthology.orchestra.run.smithi007.stdout:ID_LIKE=debian 2022-03-20T14:43:24.805 INFO:teuthology.orchestra.run.smithi007.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-20T14:43:24.806 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_ID="20.04" 2022-03-20T14:43:24.806 INFO:teuthology.orchestra.run.smithi007.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-20T14:43:24.806 INFO:teuthology.orchestra.run.smithi007.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-20T14:43:24.806 INFO:teuthology.orchestra.run.smithi007.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-20T14:43:24.807 INFO:teuthology.orchestra.run.smithi007.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-20T14:43:24.807 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_CODENAME=focal 2022-03-20T14:43:24.807 INFO:teuthology.orchestra.run.smithi007.stdout:UBUNTU_CODENAME=focal 2022-03-20T14:43:24.808 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server 2022-03-20T14:43:24.827 DEBUG:teuthology.orchestra.run.smithi057:> uname -m 2022-03-20T14:43:24.835 INFO:teuthology.orchestra.run.smithi057.stdout:x86_64 2022-03-20T14:43:24.835 DEBUG:teuthology.orchestra.run.smithi057:> cat /etc/os-release 2022-03-20T14:43:24.881 INFO:teuthology.orchestra.run.smithi057.stdout:NAME="Ubuntu" 2022-03-20T14:43:24.882 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-20T14:43:24.882 INFO:teuthology.orchestra.run.smithi057.stdout:ID=ubuntu 2022-03-20T14:43:24.882 INFO:teuthology.orchestra.run.smithi057.stdout:ID_LIKE=debian 2022-03-20T14:43:24.883 INFO:teuthology.orchestra.run.smithi057.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-20T14:43:24.883 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION_ID="20.04" 2022-03-20T14:43:24.883 INFO:teuthology.orchestra.run.smithi057.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-20T14:43:24.883 INFO:teuthology.orchestra.run.smithi057.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-20T14:43:24.883 INFO:teuthology.orchestra.run.smithi057.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-20T14:43:24.884 INFO:teuthology.orchestra.run.smithi057.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-20T14:43:24.884 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION_CODENAME=focal 2022-03-20T14:43:24.884 INFO:teuthology.orchestra.run.smithi057.stdout:UBUNTU_CODENAME=focal 2022-03-20T14:43:24.884 INFO:teuthology.lock.ops:Updating smithi057.front.sepia.ceph.com on lock server 2022-03-20T14:43:24.902 DEBUG:teuthology.orchestra.run.smithi087:> uname -m 2022-03-20T14:43:24.909 INFO:teuthology.orchestra.run.smithi087.stdout:x86_64 2022-03-20T14:43:24.909 DEBUG:teuthology.orchestra.run.smithi087:> cat /etc/os-release 2022-03-20T14:43:24.958 INFO:teuthology.orchestra.run.smithi087.stdout:NAME="Ubuntu" 2022-03-20T14:43:24.959 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-20T14:43:24.959 INFO:teuthology.orchestra.run.smithi087.stdout:ID=ubuntu 2022-03-20T14:43:24.959 INFO:teuthology.orchestra.run.smithi087.stdout:ID_LIKE=debian 2022-03-20T14:43:24.959 INFO:teuthology.orchestra.run.smithi087.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-20T14:43:24.960 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION_ID="20.04" 2022-03-20T14:43:24.960 INFO:teuthology.orchestra.run.smithi087.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-20T14:43:24.960 INFO:teuthology.orchestra.run.smithi087.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-20T14:43:24.960 INFO:teuthology.orchestra.run.smithi087.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-20T14:43:24.960 INFO:teuthology.orchestra.run.smithi087.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-20T14:43:24.961 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION_CODENAME=focal 2022-03-20T14:43:24.961 INFO:teuthology.orchestra.run.smithi087.stdout:UBUNTU_CODENAME=focal 2022-03-20T14:43:24.961 INFO:teuthology.lock.ops:Updating smithi087.front.sepia.ceph.com on lock server 2022-03-20T14:43:24.979 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-03-20T14:43:24.990 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-03-20T14:43:25.024 INFO:teuthology.task.internal:Checking for old test directory... 2022-03-20T14:43:25.024 DEBUG:teuthology.orchestra.run.smithi007:> test '!' -e /home/ubuntu/cephtest 2022-03-20T14:43:25.027 DEBUG:teuthology.orchestra.run.smithi057:> test '!' -e /home/ubuntu/cephtest 2022-03-20T14:43:25.031 DEBUG:teuthology.orchestra.run.smithi087:> test '!' -e /home/ubuntu/cephtest 2022-03-20T14:43:25.038 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-03-20T14:43:25.050 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-03-20T14:43:25.050 DEBUG:teuthology.orchestra.run.smithi007:> test -z $(ls -A /var/lib/ceph) 2022-03-20T14:43:25.052 DEBUG:teuthology.orchestra.run.smithi057:> test -z $(ls -A /var/lib/ceph) 2022-03-20T14:43:25.077 DEBUG:teuthology.orchestra.run.smithi087:> test -z $(ls -A /var/lib/ceph) 2022-03-20T14:43:25.087 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-03-20T14:43:25.164 INFO:teuthology.run_tasks:Running task kernel... 2022-03-20T14:43:25.181 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-03-20T14:43:25.183 INFO:teuthology.task.kernel:normalize config orig: {'hwe': True} 2022-03-20T14:43:25.183 DEBUG:teuthology.task.kernel:normalized overrides {'host.a': {'hwe': True}, 'client.a': {'hwe': True}, 'host.b': {'hwe': True}, 'client.b': {'hwe': True}, 'host.c': {'hwe': True}, 'client.c': {'hwe': True}} 2022-03-20T14:43:25.184 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.b': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.c': {'kdb': True, 'sha1': 'distro', 'hwe': True}}, timeout 300 2022-03-20T14:43:25.184 DEBUG:teuthology.orchestra.run.smithi007:> test -f /run/.containerenv -o -f /.dockerenv 2022-03-20T14:43:25.186 DEBUG:teuthology.orchestra.run.smithi057:> test -f /run/.containerenv -o -f /.dockerenv 2022-03-20T14:43:25.186 DEBUG:teuthology.orchestra.run.smithi087:> test -f /run/.containerenv -o -f /.dockerenv 2022-03-20T14:43:25.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-20T14:43:25.194 DEBUG:teuthology.orchestra.run.smithi057:> uname -r 2022-03-20T14:43:25.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-20T14:43:25.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-20T14:43:25.198 DEBUG:teuthology.orchestra.run.smithi087:> uname -r 2022-03-20T14:43:25.199 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2022-03-20T14:43:25.206 INFO:teuthology.orchestra.run.smithi087.stdout:5.4.0-97-generic 2022-03-20T14:43:25.206 INFO:teuthology.task.kernel:Running kernel on smithi087: 5.4.0-97-generic 2022-03-20T14:43:25.207 DEBUG:teuthology.orchestra.run.smithi087:> sudo apt-get clean 2022-03-20T14:43:25.242 INFO:teuthology.orchestra.run.smithi057.stdout:5.4.0-97-generic 2022-03-20T14:43:25.242 INFO:teuthology.task.kernel:Running kernel on smithi057: 5.4.0-97-generic 2022-03-20T14:43:25.243 DEBUG:teuthology.orchestra.run.smithi057:> sudo apt-get clean 2022-03-20T14:43:25.249 INFO:teuthology.orchestra.run.smithi007.stdout:5.4.0-97-generic 2022-03-20T14:43:25.249 INFO:teuthology.task.kernel:Running kernel on smithi007: 5.4.0-97-generic 2022-03-20T14:43:25.249 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt-get clean 2022-03-20T14:43:25.306 DEBUG:teuthology.orchestra.run.smithi057:> sudo apt-get update 2022-03-20T14:43:25.338 DEBUG:teuthology.orchestra.run.smithi087:> sudo apt-get update 2022-03-20T14:43:25.344 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt-get update 2022-03-20T14:43:25.534 INFO:teuthology.orchestra.run.smithi057.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-20T14:43:25.541 INFO:teuthology.orchestra.run.smithi087.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-20T14:43:25.551 INFO:teuthology.orchestra.run.smithi087.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-20T14:43:25.621 INFO:teuthology.orchestra.run.smithi087.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-20T14:43:25.641 INFO:teuthology.orchestra.run.smithi057.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-20T14:43:25.642 INFO:teuthology.orchestra.run.smithi087.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-20T14:43:25.657 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-20T14:43:25.658 INFO:teuthology.orchestra.run.smithi007.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-20T14:43:25.722 INFO:teuthology.orchestra.run.smithi007.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-20T14:43:25.722 INFO:teuthology.orchestra.run.smithi057.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-20T14:43:25.842 INFO:teuthology.orchestra.run.smithi057.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [402 kB] 2022-03-20T14:43:25.918 INFO:teuthology.orchestra.run.smithi087.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,642 kB] 2022-03-20T14:43:25.924 INFO:teuthology.orchestra.run.smithi057.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,317 kB] 2022-03-20T14:43:25.963 INFO:teuthology.orchestra.run.smithi057.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [231 kB] 2022-03-20T14:43:25.978 INFO:teuthology.orchestra.run.smithi057.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,804 B] 2022-03-20T14:43:25.978 INFO:teuthology.orchestra.run.smithi057.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [799 kB] 2022-03-20T14:43:25.982 INFO:teuthology.orchestra.run.smithi057.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-20T14:43:25.983 INFO:teuthology.orchestra.run.smithi057.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [114 kB] 2022-03-20T14:43:25.985 INFO:teuthology.orchestra.run.smithi057.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-20T14:43:25.985 INFO:teuthology.orchestra.run.smithi057.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [692 kB] 2022-03-20T14:43:25.997 INFO:teuthology.orchestra.run.smithi057.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [546 kB] 2022-03-20T14:43:26.006 INFO:teuthology.orchestra.run.smithi057.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [121 kB] 2022-03-20T14:43:26.008 INFO:teuthology.orchestra.run.smithi057.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.0 kB] 2022-03-20T14:43:26.020 INFO:teuthology.orchestra.run.smithi057.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-20T14:43:26.020 INFO:teuthology.orchestra.run.smithi057.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-20T14:43:26.021 INFO:teuthology.orchestra.run.smithi057.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-20T14:43:26.021 INFO:teuthology.orchestra.run.smithi057.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-20T14:43:26.049 INFO:teuthology.orchestra.run.smithi057.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-20T14:43:26.054 INFO:teuthology.orchestra.run.smithi007.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-20T14:43:26.060 INFO:teuthology.orchestra.run.smithi087.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [617 kB] 2022-03-20T14:43:26.075 INFO:teuthology.orchestra.run.smithi087.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [312 kB] 2022-03-20T14:43:26.077 INFO:teuthology.orchestra.run.smithi087.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-20T14:43:26.077 INFO:teuthology.orchestra.run.smithi087.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [853 kB] 2022-03-20T14:43:26.088 INFO:teuthology.orchestra.run.smithi087.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-20T14:43:26.089 INFO:teuthology.orchestra.run.smithi087.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [122 kB] 2022-03-20T14:43:26.091 INFO:teuthology.orchestra.run.smithi087.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-20T14:43:26.092 INFO:teuthology.orchestra.run.smithi087.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [673 kB] 2022-03-20T14:43:26.103 INFO:teuthology.orchestra.run.smithi087.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [910 kB] 2022-03-20T14:43:26.115 INFO:teuthology.orchestra.run.smithi087.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [202 kB] 2022-03-20T14:43:26.118 INFO:teuthology.orchestra.run.smithi087.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.3 kB] 2022-03-20T14:43:26.120 INFO:teuthology.orchestra.run.smithi087.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.7 kB] 2022-03-20T14:43:26.121 INFO:teuthology.orchestra.run.smithi087.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,432 B] 2022-03-20T14:43:26.121 INFO:teuthology.orchestra.run.smithi087.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-20T14:43:26.121 INFO:teuthology.orchestra.run.smithi087.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-20T14:43:26.122 INFO:teuthology.orchestra.run.smithi087.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-20T14:43:26.124 INFO:teuthology.orchestra.run.smithi087.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-20T14:43:26.135 INFO:teuthology.orchestra.run.smithi007.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,317 kB] 2022-03-20T14:43:26.144 INFO:teuthology.orchestra.run.smithi087.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-20T14:43:26.145 INFO:teuthology.orchestra.run.smithi087.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-20T14:43:26.145 INFO:teuthology.orchestra.run.smithi087.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-20T14:43:26.146 INFO:teuthology.orchestra.run.smithi087.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-20T14:43:26.146 INFO:teuthology.orchestra.run.smithi087.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-20T14:43:26.188 INFO:teuthology.orchestra.run.smithi057.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,642 kB] 2022-03-20T14:43:26.236 INFO:teuthology.orchestra.run.smithi087.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [402 kB] 2022-03-20T14:43:26.275 INFO:teuthology.orchestra.run.smithi007.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,642 kB] 2022-03-20T14:43:26.490 INFO:teuthology.orchestra.run.smithi087.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,317 kB] 2022-03-20T14:43:26.536 INFO:teuthology.orchestra.run.smithi007.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [402 kB] 2022-03-20T14:43:26.539 INFO:teuthology.orchestra.run.smithi057.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [617 kB] 2022-03-20T14:43:26.552 INFO:teuthology.orchestra.run.smithi057.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [312 kB] 2022-03-20T14:43:26.553 INFO:teuthology.orchestra.run.smithi007.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [231 kB] 2022-03-20T14:43:26.558 INFO:teuthology.orchestra.run.smithi057.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-20T14:43:26.558 INFO:teuthology.orchestra.run.smithi057.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [853 kB] 2022-03-20T14:43:26.563 INFO:teuthology.orchestra.run.smithi007.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,804 B] 2022-03-20T14:43:26.564 INFO:teuthology.orchestra.run.smithi007.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [799 kB] 2022-03-20T14:43:26.617 INFO:teuthology.orchestra.run.smithi007.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-20T14:43:26.618 INFO:teuthology.orchestra.run.smithi007.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [617 kB] 2022-03-20T14:43:26.619 INFO:teuthology.orchestra.run.smithi007.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [114 kB] 2022-03-20T14:43:26.620 INFO:teuthology.orchestra.run.smithi057.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-20T14:43:26.621 INFO:teuthology.orchestra.run.smithi057.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [122 kB] 2022-03-20T14:43:26.623 INFO:teuthology.orchestra.run.smithi007.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-20T14:43:26.624 INFO:teuthology.orchestra.run.smithi007.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [692 kB] 2022-03-20T14:43:26.625 INFO:teuthology.orchestra.run.smithi057.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-20T14:43:26.625 INFO:teuthology.orchestra.run.smithi057.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [673 kB] 2022-03-20T14:43:26.630 INFO:teuthology.orchestra.run.smithi007.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [312 kB] 2022-03-20T14:43:26.633 INFO:teuthology.orchestra.run.smithi007.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-20T14:43:26.633 INFO:teuthology.orchestra.run.smithi007.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-20T14:43:26.634 INFO:teuthology.orchestra.run.smithi057.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [910 kB] 2022-03-20T14:43:26.634 INFO:teuthology.orchestra.run.smithi007.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [853 kB] 2022-03-20T14:43:26.638 INFO:teuthology.orchestra.run.smithi087.stdout:Get:30 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [231 kB] 2022-03-20T14:43:26.642 INFO:teuthology.orchestra.run.smithi057.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [202 kB] 2022-03-20T14:43:26.647 INFO:teuthology.orchestra.run.smithi057.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.3 kB] 2022-03-20T14:43:26.653 INFO:teuthology.orchestra.run.smithi007.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [546 kB] 2022-03-20T14:43:26.656 INFO:teuthology.orchestra.run.smithi087.stdout:Get:31 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,804 B] 2022-03-20T14:43:26.656 INFO:teuthology.orchestra.run.smithi087.stdout:Get:32 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [799 kB] 2022-03-20T14:43:26.673 INFO:teuthology.orchestra.run.smithi007.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [121 kB] 2022-03-20T14:43:26.675 INFO:teuthology.orchestra.run.smithi007.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.0 kB] 2022-03-20T14:43:26.691 INFO:teuthology.orchestra.run.smithi087.stdout:Get:33 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-20T14:43:26.692 INFO:teuthology.orchestra.run.smithi087.stdout:Get:34 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [114 kB] 2022-03-20T14:43:26.696 INFO:teuthology.orchestra.run.smithi087.stdout:Get:35 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-20T14:43:26.696 INFO:teuthology.orchestra.run.smithi087.stdout:Get:36 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [546 kB] 2022-03-20T14:43:26.700 INFO:teuthology.orchestra.run.smithi007.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [122 kB] 2022-03-20T14:43:26.703 INFO:teuthology.orchestra.run.smithi007.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-20T14:43:26.703 INFO:teuthology.orchestra.run.smithi007.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [910 kB] 2022-03-20T14:43:26.706 INFO:teuthology.orchestra.run.smithi057.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,432 B] 2022-03-20T14:43:26.707 INFO:teuthology.orchestra.run.smithi057.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.7 kB] 2022-03-20T14:43:26.707 INFO:teuthology.orchestra.run.smithi057.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-20T14:43:26.708 INFO:teuthology.orchestra.run.smithi057.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-20T14:43:26.708 INFO:teuthology.orchestra.run.smithi057.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-20T14:43:26.713 INFO:teuthology.orchestra.run.smithi057.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-20T14:43:26.716 INFO:teuthology.orchestra.run.smithi007.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [673 kB] 2022-03-20T14:43:26.716 INFO:teuthology.orchestra.run.smithi007.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-20T14:43:26.717 INFO:teuthology.orchestra.run.smithi007.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-20T14:43:26.717 INFO:teuthology.orchestra.run.smithi007.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-20T14:43:26.718 INFO:teuthology.orchestra.run.smithi007.stdout:Get:30 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-20T14:43:26.718 INFO:teuthology.orchestra.run.smithi087.stdout:Get:37 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [692 kB] 2022-03-20T14:43:26.729 INFO:teuthology.orchestra.run.smithi007.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [202 kB] 2022-03-20T14:43:26.731 INFO:teuthology.orchestra.run.smithi007.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.3 kB] 2022-03-20T14:43:26.743 INFO:teuthology.orchestra.run.smithi087.stdout:Get:38 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [121 kB] 2022-03-20T14:43:26.746 INFO:teuthology.orchestra.run.smithi087.stdout:Get:39 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.0 kB] 2022-03-20T14:43:26.785 INFO:teuthology.orchestra.run.smithi007.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,432 B] 2022-03-20T14:43:26.785 INFO:teuthology.orchestra.run.smithi007.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.7 kB] 2022-03-20T14:43:26.786 INFO:teuthology.orchestra.run.smithi007.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-20T14:43:26.787 INFO:teuthology.orchestra.run.smithi007.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-20T14:43:26.787 INFO:teuthology.orchestra.run.smithi007.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-20T14:43:26.791 INFO:teuthology.orchestra.run.smithi007.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-20T14:43:26.793 INFO:teuthology.orchestra.run.smithi057.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-20T14:43:26.793 INFO:teuthology.orchestra.run.smithi057.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-20T14:43:26.794 INFO:teuthology.orchestra.run.smithi057.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-20T14:43:26.794 INFO:teuthology.orchestra.run.smithi057.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-20T14:43:26.795 INFO:teuthology.orchestra.run.smithi057.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-20T14:43:26.798 INFO:teuthology.orchestra.run.smithi087.stdout:Get:40 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-20T14:43:26.799 INFO:teuthology.orchestra.run.smithi087.stdout:Get:41 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-20T14:43:26.799 INFO:teuthology.orchestra.run.smithi087.stdout:Get:42 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-20T14:43:26.800 INFO:teuthology.orchestra.run.smithi087.stdout:Get:43 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-20T14:43:26.871 INFO:teuthology.orchestra.run.smithi007.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-20T14:43:26.872 INFO:teuthology.orchestra.run.smithi007.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-20T14:43:26.872 INFO:teuthology.orchestra.run.smithi007.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-20T14:43:26.873 INFO:teuthology.orchestra.run.smithi007.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-20T14:43:26.873 INFO:teuthology.orchestra.run.smithi007.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-20T14:43:29.196 INFO:teuthology.orchestra.run.smithi057.stdout:Fetched 10.2 MB in 2s (5,946 kB/s) 2022-03-20T14:43:29.234 INFO:teuthology.orchestra.run.smithi087.stdout:Fetched 10.2 MB in 2s (5,807 kB/s) 2022-03-20T14:43:29.762 INFO:teuthology.orchestra.run.smithi007.stdout:Fetched 10.2 MB in 2s (4,675 kB/s) 2022-03-20T14:43:30.401 INFO:teuthology.orchestra.run.smithi057.stdout:Reading package lists... 2022-03-20T14:43:30.424 DEBUG:teuthology.orchestra.run.smithi057:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-20T14:43:30.440 INFO:teuthology.orchestra.run.smithi087.stdout:Reading package lists... 2022-03-20T14:43:30.462 DEBUG:teuthology.orchestra.run.smithi087:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-20T14:43:30.501 INFO:teuthology.orchestra.run.smithi057.stdout:Reading package lists... 2022-03-20T14:43:30.531 INFO:teuthology.orchestra.run.smithi087.stdout:Reading package lists... 2022-03-20T14:43:30.676 INFO:teuthology.orchestra.run.smithi057.stdout:Building dependency tree... 2022-03-20T14:43:30.677 INFO:teuthology.orchestra.run.smithi057.stdout:Reading state information... 2022-03-20T14:43:30.706 INFO:teuthology.orchestra.run.smithi087.stdout:Building dependency tree... 2022-03-20T14:43:30.706 INFO:teuthology.orchestra.run.smithi087.stdout:Reading state information... 2022-03-20T14:43:30.818 INFO:teuthology.orchestra.run.smithi057.stdout:The following additional packages will be installed: 2022-03-20T14:43:30.818 INFO:teuthology.orchestra.run.smithi057.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-20T14:43:30.818 INFO:teuthology.orchestra.run.smithi057.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-modules-5.13.0-35-generic 2022-03-20T14:43:30.819 INFO:teuthology.orchestra.run.smithi057.stdout: linux-modules-extra-5.13.0-35-generic thermald upower usbmuxd 2022-03-20T14:43:30.819 INFO:teuthology.orchestra.run.smithi057.stdout:Suggested packages: 2022-03-20T14:43:30.819 INFO:teuthology.orchestra.run.smithi057.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-20T14:43:30.820 INFO:teuthology.orchestra.run.smithi057.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-35-generic 2022-03-20T14:43:30.856 INFO:teuthology.orchestra.run.smithi087.stdout:The following additional packages will be installed: 2022-03-20T14:43:30.856 INFO:teuthology.orchestra.run.smithi087.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-20T14:43:30.856 INFO:teuthology.orchestra.run.smithi087.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-modules-5.13.0-35-generic 2022-03-20T14:43:30.857 INFO:teuthology.orchestra.run.smithi087.stdout: linux-modules-extra-5.13.0-35-generic thermald upower usbmuxd 2022-03-20T14:43:30.857 INFO:teuthology.orchestra.run.smithi087.stdout:Suggested packages: 2022-03-20T14:43:30.858 INFO:teuthology.orchestra.run.smithi087.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-20T14:43:30.858 INFO:teuthology.orchestra.run.smithi087.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-35-generic 2022-03-20T14:43:30.865 INFO:teuthology.orchestra.run.smithi057.stdout:The following NEW packages will be installed: 2022-03-20T14:43:30.866 INFO:teuthology.orchestra.run.smithi057.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-20T14:43:30.866 INFO:teuthology.orchestra.run.smithi057.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-image-generic-hwe-20.04 2022-03-20T14:43:30.867 INFO:teuthology.orchestra.run.smithi057.stdout: linux-modules-5.13.0-35-generic linux-modules-extra-5.13.0-35-generic 2022-03-20T14:43:30.867 INFO:teuthology.orchestra.run.smithi057.stdout: thermald upower usbmuxd 2022-03-20T14:43:30.910 INFO:teuthology.orchestra.run.smithi087.stdout:The following NEW packages will be installed: 2022-03-20T14:43:30.911 INFO:teuthology.orchestra.run.smithi087.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-20T14:43:30.911 INFO:teuthology.orchestra.run.smithi087.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-image-generic-hwe-20.04 2022-03-20T14:43:30.911 INFO:teuthology.orchestra.run.smithi087.stdout: linux-modules-5.13.0-35-generic linux-modules-extra-5.13.0-35-generic 2022-03-20T14:43:30.912 INFO:teuthology.orchestra.run.smithi087.stdout: thermald upower usbmuxd 2022-03-20T14:43:30.980 INFO:teuthology.orchestra.run.smithi087.stdout:0 upgraded, 13 newly installed, 0 to remove and 109 not upgraded. 2022-03-20T14:43:30.980 INFO:teuthology.orchestra.run.smithi087.stdout:Need to get 80.1 MB of archives. 2022-03-20T14:43:30.980 INFO:teuthology.orchestra.run.smithi087.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-20T14:43:30.980 INFO:teuthology.orchestra.run.smithi087.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-20T14:43:31.001 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2022-03-20T14:43:31.024 DEBUG:teuthology.orchestra.run.smithi007:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-20T14:43:31.044 INFO:teuthology.orchestra.run.smithi057.stdout:0 upgraded, 13 newly installed, 0 to remove and 109 not upgraded. 2022-03-20T14:43:31.044 INFO:teuthology.orchestra.run.smithi057.stdout:Need to get 80.1 MB of archives. 2022-03-20T14:43:31.044 INFO:teuthology.orchestra.run.smithi057.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-20T14:43:31.044 INFO:teuthology.orchestra.run.smithi057.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-20T14:43:31.055 INFO:teuthology.orchestra.run.smithi087.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-20T14:43:31.056 INFO:teuthology.orchestra.run.smithi087.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-20T14:43:31.057 INFO:teuthology.orchestra.run.smithi087.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-20T14:43:31.084 INFO:teuthology.orchestra.run.smithi087.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-20T14:43:31.089 INFO:teuthology.orchestra.run.smithi087.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [18.3 MB] 2022-03-20T14:43:31.092 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2022-03-20T14:43:31.269 INFO:teuthology.orchestra.run.smithi007.stdout:Building dependency tree... 2022-03-20T14:43:31.270 INFO:teuthology.orchestra.run.smithi007.stdout:Reading state information... 2022-03-20T14:43:31.303 INFO:teuthology.orchestra.run.smithi057.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-20T14:43:31.306 INFO:teuthology.orchestra.run.smithi057.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-20T14:43:31.322 INFO:teuthology.orchestra.run.smithi057.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-20T14:43:31.400 INFO:teuthology.orchestra.run.smithi057.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-20T14:43:31.401 INFO:teuthology.orchestra.run.smithi057.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [18.3 MB] 2022-03-20T14:43:31.423 INFO:teuthology.orchestra.run.smithi007.stdout:The following additional packages will be installed: 2022-03-20T14:43:31.423 INFO:teuthology.orchestra.run.smithi007.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-20T14:43:31.423 INFO:teuthology.orchestra.run.smithi007.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-modules-5.13.0-35-generic 2022-03-20T14:43:31.424 INFO:teuthology.orchestra.run.smithi007.stdout: linux-modules-extra-5.13.0-35-generic thermald upower usbmuxd 2022-03-20T14:43:31.424 INFO:teuthology.orchestra.run.smithi007.stdout:Suggested packages: 2022-03-20T14:43:31.424 INFO:teuthology.orchestra.run.smithi007.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-20T14:43:31.425 INFO:teuthology.orchestra.run.smithi007.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-35-generic 2022-03-20T14:43:31.466 INFO:teuthology.orchestra.run.smithi007.stdout:The following NEW packages will be installed: 2022-03-20T14:43:31.467 INFO:teuthology.orchestra.run.smithi007.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-20T14:43:31.468 INFO:teuthology.orchestra.run.smithi007.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-image-generic-hwe-20.04 2022-03-20T14:43:31.468 INFO:teuthology.orchestra.run.smithi007.stdout: linux-modules-5.13.0-35-generic linux-modules-extra-5.13.0-35-generic 2022-03-20T14:43:31.468 INFO:teuthology.orchestra.run.smithi007.stdout: thermald upower usbmuxd 2022-03-20T14:43:31.525 INFO:teuthology.orchestra.run.smithi087.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [10.0 MB] 2022-03-20T14:43:31.648 INFO:teuthology.orchestra.run.smithi007.stdout:0 upgraded, 13 newly installed, 0 to remove and 109 not upgraded. 2022-03-20T14:43:31.649 INFO:teuthology.orchestra.run.smithi007.stdout:Need to get 80.1 MB of archives. 2022-03-20T14:43:31.649 INFO:teuthology.orchestra.run.smithi007.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-20T14:43:31.649 INFO:teuthology.orchestra.run.smithi007.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-20T14:43:31.654 INFO:teuthology.orchestra.run.smithi087.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [51.1 MB] 2022-03-20T14:43:31.907 INFO:teuthology.orchestra.run.smithi007.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-20T14:43:31.934 INFO:teuthology.orchestra.run.smithi007.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-20T14:43:31.955 INFO:teuthology.orchestra.run.smithi007.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-20T14:43:32.031 INFO:teuthology.orchestra.run.smithi007.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-20T14:43:32.052 INFO:teuthology.orchestra.run.smithi007.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [18.3 MB] 2022-03-20T14:43:32.277 INFO:teuthology.orchestra.run.smithi057.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [10.0 MB] 2022-03-20T14:43:32.532 INFO:teuthology.orchestra.run.smithi057.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [51.1 MB] 2022-03-20T14:43:32.678 INFO:teuthology.orchestra.run.smithi087.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.35.40~20.04.20 [2,632 B] 2022-03-20T14:43:32.678 INFO:teuthology.orchestra.run.smithi087.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-20T14:43:32.679 INFO:teuthology.orchestra.run.smithi087.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-20T14:43:32.683 INFO:teuthology.orchestra.run.smithi087.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-20T14:43:32.685 INFO:teuthology.orchestra.run.smithi087.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-20T14:43:32.849 INFO:teuthology.orchestra.run.smithi007.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [10.0 MB] 2022-03-20T14:43:33.084 INFO:teuthology.orchestra.run.smithi087.stdout:Fetched 80.1 MB in 2s (45.3 MB/s) 2022-03-20T14:43:33.105 INFO:teuthology.orchestra.run.smithi007.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [51.1 MB] 2022-03-20T14:43:33.306 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-20T14:43:34.223 INFO:teuthology.orchestra.run.smithi057.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.35.40~20.04.20 [2,632 B] 2022-03-20T14:43:34.223 INFO:teuthology.orchestra.run.smithi057.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-20T14:43:34.224 INFO:teuthology.orchestra.run.smithi057.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-20T14:43:34.230 INFO:teuthology.orchestra.run.smithi057.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-20T14:43:34.232 INFO:teuthology.orchestra.run.smithi057.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-20T14:43:34.661 INFO:teuthology.orchestra.run.smithi057.stdout:Fetched 80.1 MB in 3s (23.8 MB/s) 2022-03-20T14:43:34.709 INFO:teuthology.orchestra.run.smithi007.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.35.40~20.04.20 [2,632 B] 2022-03-20T14:43:34.710 INFO:teuthology.orchestra.run.smithi007.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-20T14:43:34.710 INFO:teuthology.orchestra.run.smithi007.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-20T14:43:34.756 INFO:teuthology.orchestra.run.smithi087.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-20T14:43:34.762 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-20T14:43:34.773 INFO:teuthology.orchestra.run.smithi007.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-20T14:43:34.774 INFO:teuthology.orchestra.run.smithi007.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-20T14:43:34.849 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-20T14:43:34.877 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-20T14:43:35.195 INFO:teuthology.orchestra.run.smithi007.stdout:Fetched 80.1 MB in 3s (24.3 MB/s) 2022-03-20T14:43:35.239 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-20T14:43:35.253 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-20T14:43:35.297 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-20T14:43:35.399 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-20T14:43:35.566 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-20T14:43:35.579 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-20T14:43:35.616 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-20T14:43:35.953 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-20T14:43:35.968 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-20T14:43:36.011 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-20T14:43:36.343 INFO:teuthology.orchestra.run.smithi057.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-20T14:43:36.349 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-20T14:43:36.355 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-20T14:43:36.369 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-20T14:43:36.413 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-20T14:43:36.472 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-20T14:43:36.691 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package linux-modules-5.13.0-35-generic. 2022-03-20T14:43:36.706 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../05-linux-modules-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-20T14:43:36.741 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:43:36.833 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-20T14:43:36.848 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-20T14:43:36.892 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-20T14:43:36.901 INFO:teuthology.orchestra.run.smithi007.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-20T14:43:36.906 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-20T14:43:37.046 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-20T14:43:37.228 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-20T14:43:37.241 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-20T14:43:37.278 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-20T14:43:37.424 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-20T14:43:37.438 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-20T14:43:37.482 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-20T14:43:37.606 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-20T14:43:37.620 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-20T14:43:37.656 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-20T14:43:37.811 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-20T14:43:37.823 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-20T14:43:37.860 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-20T14:43:38.001 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-20T14:43:38.017 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-20T14:43:38.059 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-20T14:43:38.204 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-20T14:43:38.220 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-20T14:43:38.263 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-20T14:43:38.379 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package linux-modules-5.13.0-35-generic. 2022-03-20T14:43:38.392 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../05-linux-modules-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-20T14:43:38.429 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:43:38.616 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-20T14:43:38.630 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-20T14:43:38.683 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-20T14:43:38.915 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package linux-image-5.13.0-35-generic. 2022-03-20T14:43:38.931 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../06-linux-image-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-20T14:43:38.986 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package linux-modules-5.13.0-35-generic. 2022-03-20T14:43:39.001 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../05-linux-modules-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-20T14:43:39.031 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:43:39.044 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:43:39.502 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-35-generic. 2022-03-20T14:43:39.518 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-20T14:43:39.560 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:43:40.765 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package linux-image-5.13.0-35-generic. 2022-03-20T14:43:40.780 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../06-linux-image-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-20T14:43:40.873 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:43:41.313 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package linux-image-5.13.0-35-generic. 2022-03-20T14:43:41.328 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../06-linux-image-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-20T14:43:41.368 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-35-generic. 2022-03-20T14:43:41.381 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-20T14:43:41.440 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:43:41.442 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:43:41.932 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-35-generic. 2022-03-20T14:43:41.948 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-20T14:43:41.990 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:43:45.440 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-20T14:43:45.457 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.35.40~20.04.20_amd64.deb ... 2022-03-20T14:43:45.498 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-20T14:43:45.814 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-20T14:43:45.829 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-20T14:43:45.864 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-20T14:43:46.242 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package thermald. 2022-03-20T14:43:46.257 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-20T14:43:46.300 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-20T14:43:46.850 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package upower. 2022-03-20T14:43:46.864 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-20T14:43:46.917 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-20T14:43:47.316 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package usbmuxd. 2022-03-20T14:43:47.329 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-20T14:43:47.361 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-20T14:43:47.374 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-20T14:43:47.376 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.35.40~20.04.20_amd64.deb ... 2022-03-20T14:43:47.419 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-20T14:43:47.682 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-20T14:43:47.734 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-20T14:43:47.743 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-20T14:43:47.779 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-20T14:43:47.804 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-20T14:43:47.836 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-20T14:43:47.851 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.35.40~20.04.20_amd64.deb ... 2022-03-20T14:43:47.894 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-20T14:43:47.922 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-20T14:43:48.040 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-20T14:43:48.107 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package thermald. 2022-03-20T14:43:48.121 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-20T14:43:48.165 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-20T14:43:48.166 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-20T14:43:48.292 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-20T14:43:48.355 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-20T14:43:48.370 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-20T14:43:48.418 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-20T14:43:48.422 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-20T14:43:48.722 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package upower. 2022-03-20T14:43:48.736 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-20T14:43:48.791 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-20T14:43:48.792 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package thermald. 2022-03-20T14:43:48.806 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-20T14:43:48.850 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-20T14:43:48.865 INFO:teuthology.orchestra.run.smithi087.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-20T14:43:48.920 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-20T14:43:49.096 INFO:teuthology.orchestra.run.smithi087.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-20T14:43:49.105 INFO:teuthology.orchestra.run.smithi087.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-20T14:43:49.105 INFO:teuthology.orchestra.run.smithi087.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-20T14:43:49.195 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package usbmuxd. 2022-03-20T14:43:49.209 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-20T14:43:49.270 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-20T14:43:49.391 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package upower. 2022-03-20T14:43:49.408 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-20T14:43:49.476 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-20T14:43:49.561 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-20T14:43:49.578 INFO:teuthology.orchestra.run.smithi087.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-20T14:43:49.685 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-20T14:43:49.800 INFO:teuthology.orchestra.run.smithi087.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-20T14:43:49.811 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-20T14:43:49.882 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package usbmuxd. 2022-03-20T14:43:49.891 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-20T14:43:49.895 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-20T14:43:49.937 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-20T14:43:49.941 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-20T14:43:50.207 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-20T14:43:50.207 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-20T14:43:50.255 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-20T14:43:50.380 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-20T14:43:50.471 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-20T14:43:50.523 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-20T14:43:50.632 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-20T14:43:50.656 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-20T14:43:50.662 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:43:50.742 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-20T14:43:50.773 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-20T14:43:50.868 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-20T14:43:51.019 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-20T14:43:51.240 INFO:teuthology.orchestra.run.smithi057.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-20T14:43:51.300 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-20T14:43:51.461 INFO:teuthology.orchestra.run.smithi007.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-20T14:43:51.471 INFO:teuthology.orchestra.run.smithi057.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-20T14:43:51.476 INFO:teuthology.orchestra.run.smithi057.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-20T14:43:51.476 INFO:teuthology.orchestra.run.smithi057.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-20T14:43:51.537 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-20T14:43:51.713 INFO:teuthology.orchestra.run.smithi007.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-20T14:43:51.727 INFO:teuthology.orchestra.run.smithi007.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-20T14:43:51.727 INFO:teuthology.orchestra.run.smithi007.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-20T14:43:51.921 INFO:teuthology.orchestra.run.smithi087.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-35-generic 2022-03-20T14:43:51.921 INFO:teuthology.orchestra.run.smithi087.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-35-generic 2022-03-20T14:43:51.950 INFO:teuthology.orchestra.run.smithi057.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-20T14:43:52.089 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:43:52.184 INFO:teuthology.orchestra.run.smithi057.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-20T14:43:52.229 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-20T14:43:52.481 INFO:teuthology.orchestra.run.smithi007.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-20T14:43:52.545 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-20T14:43:52.545 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-20T14:43:52.724 INFO:teuthology.orchestra.run.smithi007.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-20T14:43:52.767 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-20T14:43:53.034 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:43:53.072 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-20T14:43:53.072 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-20T14:43:53.407 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:43:53.530 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:43:54.332 INFO:teuthology.orchestra.run.smithi057.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-35-generic 2022-03-20T14:43:54.332 INFO:teuthology.orchestra.run.smithi057.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-35-generic 2022-03-20T14:43:54.469 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:43:54.709 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-20T14:43:54.818 INFO:teuthology.orchestra.run.smithi087.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-20T14:43:54.832 INFO:teuthology.orchestra.run.smithi007.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-35-generic 2022-03-20T14:43:54.832 INFO:teuthology.orchestra.run.smithi007.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-35-generic 2022-03-20T14:43:54.982 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:43:55.828 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:43:56.326 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:43:57.230 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-20T14:43:57.355 INFO:teuthology.orchestra.run.smithi057.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-20T14:43:58.178 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-20T14:43:58.328 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-20T14:43:59.123 INFO:teuthology.orchestra.run.smithi087.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-20T14:43:59.800 INFO:teuthology.orchestra.run.smithi087.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-20T14:43:59.892 INFO:teuthology.orchestra.run.smithi087.stdout:Processing triggers for linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:44:00.082 INFO:teuthology.orchestra.run.smithi087.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-20T14:44:00.083 INFO:teuthology.orchestra.run.smithi087.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-35-generic 2022-03-20T14:44:01.692 INFO:teuthology.orchestra.run.smithi057.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-20T14:44:02.242 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-20T14:44:02.896 INFO:teuthology.orchestra.run.smithi057.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-20T14:44:02.911 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-20T14:44:02.997 INFO:teuthology.orchestra.run.smithi057.stdout:Processing triggers for linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:44:03.011 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-20T14:44:03.178 INFO:teuthology.orchestra.run.smithi057.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-20T14:44:03.178 INFO:teuthology.orchestra.run.smithi057.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-35-generic 2022-03-20T14:44:03.377 INFO:teuthology.orchestra.run.smithi007.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-20T14:44:03.377 INFO:teuthology.orchestra.run.smithi007.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-35-generic 2022-03-20T14:44:23.475 INFO:teuthology.orchestra.run.smithi087.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-20T14:44:23.475 INFO:teuthology.orchestra.run.smithi087.stdout:Sourcing file `/etc/default/grub' 2022-03-20T14:44:23.486 INFO:teuthology.orchestra.run.smithi087.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-20T14:44:23.519 INFO:teuthology.orchestra.run.smithi087.stdout:Generating grub configuration file ... 2022-03-20T14:44:23.876 INFO:teuthology.orchestra.run.smithi087.stdout:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-20T14:44:23.901 INFO:teuthology.orchestra.run.smithi087.stdout:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-20T14:44:24.278 INFO:teuthology.orchestra.run.smithi087.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-20T14:44:24.290 INFO:teuthology.orchestra.run.smithi087.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-20T14:44:26.887 INFO:teuthology.orchestra.run.smithi087.stdout:done 2022-03-20T14:44:26.892 INFO:teuthology.orchestra.run.smithi057.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-20T14:44:26.892 INFO:teuthology.orchestra.run.smithi057.stdout:Sourcing file `/etc/default/grub' 2022-03-20T14:44:26.897 INFO:teuthology.orchestra.run.smithi057.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-20T14:44:26.921 INFO:teuthology.orchestra.run.smithi057.stdout:Generating grub configuration file ... 2022-03-20T14:44:27.211 INFO:teuthology.orchestra.run.smithi057.stdout:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-20T14:44:27.225 DEBUG:teuthology.orchestra.run.smithi087:> dpkg -s linux-image-generic-hwe-20.04 2022-03-20T14:44:27.240 INFO:teuthology.orchestra.run.smithi057.stdout:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-20T14:44:27.249 INFO:teuthology.orchestra.run.smithi087.stdout:Package: linux-image-generic-hwe-20.04 2022-03-20T14:44:27.250 INFO:teuthology.orchestra.run.smithi087.stdout:Status: install ok installed 2022-03-20T14:44:27.250 INFO:teuthology.orchestra.run.smithi087.stdout:Priority: optional 2022-03-20T14:44:27.250 INFO:teuthology.orchestra.run.smithi087.stdout:Section: kernel 2022-03-20T14:44:27.250 INFO:teuthology.orchestra.run.smithi087.stdout:Installed-Size: 19 2022-03-20T14:44:27.251 INFO:teuthology.orchestra.run.smithi087.stdout:Maintainer: Ubuntu Kernel Team 2022-03-20T14:44:27.251 INFO:teuthology.orchestra.run.smithi087.stdout:Architecture: amd64 2022-03-20T14:44:27.251 INFO:teuthology.orchestra.run.smithi087.stdout:Source: linux-meta-hwe-5.13 2022-03-20T14:44:27.251 INFO:teuthology.orchestra.run.smithi087.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-20T14:44:27.252 INFO:teuthology.orchestra.run.smithi087.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-20T14:44:27.252 INFO:teuthology.orchestra.run.smithi087.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-20T14:44:27.252 INFO:teuthology.orchestra.run.smithi087.stdout:Recommends: thermald 2022-03-20T14:44:27.252 INFO:teuthology.orchestra.run.smithi087.stdout:Description: Generic Linux kernel image 2022-03-20T14:44:27.252 INFO:teuthology.orchestra.run.smithi087.stdout: This package will always depend on the latest generic kernel image 2022-03-20T14:44:27.253 INFO:teuthology.orchestra.run.smithi087.stdout: available. 2022-03-20T14:44:27.253 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-35-generic 2022-03-20T14:44:27.253 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-20T14:44:27.254 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-20T14:44:27.254 INFO:teuthology.task.kernel:Installing distro kernel on host.c... 2022-03-20T14:44:27.254 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi087.front.sepia.ceph.com, path=None, version=distro) 2022-03-20T14:44:27.254 DEBUG:teuthology.orchestra.run.smithi087:> sudo apt-get clean 2022-03-20T14:44:27.312 DEBUG:teuthology.orchestra.run.smithi087:> sudo apt-get update 2022-03-20T14:44:27.447 INFO:teuthology.orchestra.run.smithi087.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-20T14:44:27.556 INFO:teuthology.orchestra.run.smithi087.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-20T14:44:27.636 INFO:teuthology.orchestra.run.smithi087.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-20T14:44:27.660 INFO:teuthology.orchestra.run.smithi057.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-20T14:44:27.672 INFO:teuthology.orchestra.run.smithi057.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-20T14:44:27.716 INFO:teuthology.orchestra.run.smithi087.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-20T14:44:27.731 INFO:teuthology.orchestra.run.smithi007.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-20T14:44:27.731 INFO:teuthology.orchestra.run.smithi007.stdout:Sourcing file `/etc/default/grub' 2022-03-20T14:44:27.740 INFO:teuthology.orchestra.run.smithi007.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-20T14:44:27.770 INFO:teuthology.orchestra.run.smithi007.stdout:Generating grub configuration file ... 2022-03-20T14:44:28.056 INFO:teuthology.orchestra.run.smithi007.stdout:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-20T14:44:28.106 INFO:teuthology.orchestra.run.smithi007.stdout:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-20T14:44:28.479 INFO:teuthology.orchestra.run.smithi007.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-20T14:44:28.490 INFO:teuthology.orchestra.run.smithi007.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-20T14:44:28.979 INFO:teuthology.orchestra.run.smithi087.stdout:Reading package lists... 2022-03-20T14:44:29.001 DEBUG:teuthology.orchestra.run.smithi087:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-20T14:44:29.070 INFO:teuthology.orchestra.run.smithi087.stdout:Reading package lists... 2022-03-20T14:44:29.223 INFO:teuthology.orchestra.run.smithi087.stdout:Building dependency tree... 2022-03-20T14:44:29.224 INFO:teuthology.orchestra.run.smithi087.stdout:Reading state information... 2022-03-20T14:44:29.390 INFO:teuthology.orchestra.run.smithi087.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.35.40~20.04.20). 2022-03-20T14:44:29.390 INFO:teuthology.orchestra.run.smithi087.stdout:0 upgraded, 0 newly installed, 0 to remove and 109 not upgraded. 2022-03-20T14:44:29.392 DEBUG:teuthology.orchestra.run.smithi087:> dpkg -s linux-image-generic-hwe-20.04 2022-03-20T14:44:29.408 INFO:teuthology.orchestra.run.smithi087.stdout:Package: linux-image-generic-hwe-20.04 2022-03-20T14:44:29.408 INFO:teuthology.orchestra.run.smithi087.stdout:Status: install ok installed 2022-03-20T14:44:29.408 INFO:teuthology.orchestra.run.smithi087.stdout:Priority: optional 2022-03-20T14:44:29.409 INFO:teuthology.orchestra.run.smithi087.stdout:Section: kernel 2022-03-20T14:44:29.409 INFO:teuthology.orchestra.run.smithi087.stdout:Installed-Size: 19 2022-03-20T14:44:29.409 INFO:teuthology.orchestra.run.smithi087.stdout:Maintainer: Ubuntu Kernel Team 2022-03-20T14:44:29.409 INFO:teuthology.orchestra.run.smithi087.stdout:Architecture: amd64 2022-03-20T14:44:29.409 INFO:teuthology.orchestra.run.smithi087.stdout:Source: linux-meta-hwe-5.13 2022-03-20T14:44:29.410 INFO:teuthology.orchestra.run.smithi087.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-20T14:44:29.410 INFO:teuthology.orchestra.run.smithi087.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-20T14:44:29.410 INFO:teuthology.orchestra.run.smithi087.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-20T14:44:29.410 INFO:teuthology.orchestra.run.smithi087.stdout:Recommends: thermald 2022-03-20T14:44:29.411 INFO:teuthology.orchestra.run.smithi087.stdout:Description: Generic Linux kernel image 2022-03-20T14:44:29.411 INFO:teuthology.orchestra.run.smithi087.stdout: This package will always depend on the latest generic kernel image 2022-03-20T14:44:29.411 INFO:teuthology.orchestra.run.smithi087.stdout: available. 2022-03-20T14:44:29.411 DEBUG:teuthology.orchestra.run.smithi087:> mktemp 2022-03-20T14:44:29.418 INFO:teuthology.orchestra.run.smithi087.stdout:/tmp/tmp.pgUSpnqa1V 2022-03-20T14:44:29.419 DEBUG:teuthology.orchestra.run.smithi087:> sudo cp /boot/grub/grub.cfg /tmp/tmp.pgUSpnqa1V 2022-03-20T14:44:29.472 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0666 /tmp/tmp.pgUSpnqa1V 2022-03-20T14:44:29.519 DEBUG:teuthology.orchestra.remote:smithi087:/tmp/tmp.pgUSpnqa1V is 8KB 2022-03-20T14:44:29.534 DEBUG:teuthology.orchestra.run.smithi087:> rm -fr /tmp/tmp.pgUSpnqa1V 2022-03-20T14:44:29.541 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-20T14:44:29.599 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2022-03-20T14:44:29.599 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-20T14:44:29.599 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-20T14:44:29.624 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-35-generic 2022-03-20T14:44:29.625 DEBUG:teuthology.orchestra.run.smithi087:> sudo update-grub 2022-03-20T14:44:30.363 INFO:teuthology.orchestra.run.smithi057.stdout:done 2022-03-20T14:44:30.735 DEBUG:teuthology.orchestra.run.smithi057:> dpkg -s linux-image-generic-hwe-20.04 2022-03-20T14:44:30.764 INFO:teuthology.orchestra.run.smithi057.stdout:Package: linux-image-generic-hwe-20.04 2022-03-20T14:44:30.764 INFO:teuthology.orchestra.run.smithi057.stdout:Status: install ok installed 2022-03-20T14:44:30.764 INFO:teuthology.orchestra.run.smithi057.stdout:Priority: optional 2022-03-20T14:44:30.765 INFO:teuthology.orchestra.run.smithi057.stdout:Section: kernel 2022-03-20T14:44:30.765 INFO:teuthology.orchestra.run.smithi057.stdout:Installed-Size: 19 2022-03-20T14:44:30.765 INFO:teuthology.orchestra.run.smithi057.stdout:Maintainer: Ubuntu Kernel Team 2022-03-20T14:44:30.765 INFO:teuthology.orchestra.run.smithi057.stdout:Architecture: amd64 2022-03-20T14:44:30.766 INFO:teuthology.orchestra.run.smithi057.stdout:Source: linux-meta-hwe-5.13 2022-03-20T14:44:30.766 INFO:teuthology.orchestra.run.smithi057.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-20T14:44:30.766 INFO:teuthology.orchestra.run.smithi057.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-20T14:44:30.766 INFO:teuthology.orchestra.run.smithi057.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-20T14:44:30.767 INFO:teuthology.orchestra.run.smithi057.stdout:Recommends: thermald 2022-03-20T14:44:30.767 INFO:teuthology.orchestra.run.smithi057.stdout:Description: Generic Linux kernel image 2022-03-20T14:44:30.767 INFO:teuthology.orchestra.run.smithi057.stdout: This package will always depend on the latest generic kernel image 2022-03-20T14:44:30.767 INFO:teuthology.orchestra.run.smithi057.stdout: available. 2022-03-20T14:44:30.768 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-35-generic 2022-03-20T14:44:30.768 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-20T14:44:30.768 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-20T14:44:30.768 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2022-03-20T14:44:30.769 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi057.front.sepia.ceph.com, path=None, version=distro) 2022-03-20T14:44:30.769 DEBUG:teuthology.orchestra.run.smithi057:> sudo apt-get clean 2022-03-20T14:44:30.770 INFO:teuthology.orchestra.run.smithi087.stderr:Sourcing file `/etc/default/grub' 2022-03-20T14:44:30.771 INFO:teuthology.orchestra.run.smithi087.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-20T14:44:30.785 INFO:teuthology.orchestra.run.smithi087.stderr:Generating grub configuration file ... 2022-03-20T14:44:30.824 DEBUG:teuthology.orchestra.run.smithi057:> sudo apt-get update 2022-03-20T14:44:31.018 INFO:teuthology.orchestra.run.smithi087.stderr:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-20T14:44:31.030 INFO:teuthology.orchestra.run.smithi087.stderr:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-20T14:44:31.059 INFO:teuthology.orchestra.run.smithi007.stdout:done 2022-03-20T14:44:31.102 INFO:teuthology.orchestra.run.smithi057.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-20T14:44:31.102 INFO:teuthology.orchestra.run.smithi057.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-20T14:44:31.171 INFO:teuthology.orchestra.run.smithi057.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-20T14:44:31.251 INFO:teuthology.orchestra.run.smithi057.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-20T14:44:31.376 INFO:teuthology.orchestra.run.smithi087.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-20T14:44:31.387 INFO:teuthology.orchestra.run.smithi087.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-20T14:44:31.441 DEBUG:teuthology.orchestra.run.smithi007:> dpkg -s linux-image-generic-hwe-20.04 2022-03-20T14:44:31.470 INFO:teuthology.orchestra.run.smithi007.stdout:Package: linux-image-generic-hwe-20.04 2022-03-20T14:44:31.470 INFO:teuthology.orchestra.run.smithi007.stdout:Status: install ok installed 2022-03-20T14:44:31.470 INFO:teuthology.orchestra.run.smithi007.stdout:Priority: optional 2022-03-20T14:44:31.471 INFO:teuthology.orchestra.run.smithi007.stdout:Section: kernel 2022-03-20T14:44:31.471 INFO:teuthology.orchestra.run.smithi007.stdout:Installed-Size: 19 2022-03-20T14:44:31.471 INFO:teuthology.orchestra.run.smithi007.stdout:Maintainer: Ubuntu Kernel Team 2022-03-20T14:44:31.471 INFO:teuthology.orchestra.run.smithi007.stdout:Architecture: amd64 2022-03-20T14:44:31.471 INFO:teuthology.orchestra.run.smithi007.stdout:Source: linux-meta-hwe-5.13 2022-03-20T14:44:31.472 INFO:teuthology.orchestra.run.smithi007.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-20T14:44:31.472 INFO:teuthology.orchestra.run.smithi007.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-20T14:44:31.472 INFO:teuthology.orchestra.run.smithi007.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-20T14:44:31.472 INFO:teuthology.orchestra.run.smithi007.stdout:Recommends: thermald 2022-03-20T14:44:31.473 INFO:teuthology.orchestra.run.smithi007.stdout:Description: Generic Linux kernel image 2022-03-20T14:44:31.473 INFO:teuthology.orchestra.run.smithi007.stdout: This package will always depend on the latest generic kernel image 2022-03-20T14:44:31.473 INFO:teuthology.orchestra.run.smithi007.stdout: available. 2022-03-20T14:44:31.473 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-35-generic 2022-03-20T14:44:31.474 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-20T14:44:31.474 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-20T14:44:31.474 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2022-03-20T14:44:31.474 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi007.front.sepia.ceph.com, path=None, version=distro) 2022-03-20T14:44:31.475 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt-get clean 2022-03-20T14:44:31.557 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt-get update 2022-03-20T14:44:31.708 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-20T14:44:31.725 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-20T14:44:31.754 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-20T14:44:31.805 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-20T14:44:32.701 INFO:teuthology.orchestra.run.smithi057.stdout:Reading package lists... 2022-03-20T14:44:32.725 DEBUG:teuthology.orchestra.run.smithi057:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-20T14:44:32.796 INFO:teuthology.orchestra.run.smithi057.stdout:Reading package lists... 2022-03-20T14:44:32.956 INFO:teuthology.orchestra.run.smithi057.stdout:Building dependency tree... 2022-03-20T14:44:32.957 INFO:teuthology.orchestra.run.smithi057.stdout:Reading state information... 2022-03-20T14:44:33.007 INFO:teuthology.orchestra.run.smithi087.stderr:done 2022-03-20T14:44:33.009 DEBUG:teuthology.orchestra.run.smithi087:> sudo shutdown -r now 2022-03-20T14:44:33.136 INFO:teuthology.orchestra.run.smithi057.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.35.40~20.04.20). 2022-03-20T14:44:33.136 INFO:teuthology.orchestra.run.smithi057.stdout:0 upgraded, 0 newly installed, 0 to remove and 109 not upgraded. 2022-03-20T14:44:33.138 DEBUG:teuthology.orchestra.run.smithi057:> dpkg -s linux-image-generic-hwe-20.04 2022-03-20T14:44:33.158 INFO:teuthology.orchestra.run.smithi057.stdout:Package: linux-image-generic-hwe-20.04 2022-03-20T14:44:33.158 INFO:teuthology.orchestra.run.smithi057.stdout:Status: install ok installed 2022-03-20T14:44:33.159 INFO:teuthology.orchestra.run.smithi057.stdout:Priority: optional 2022-03-20T14:44:33.159 INFO:teuthology.orchestra.run.smithi057.stdout:Section: kernel 2022-03-20T14:44:33.159 INFO:teuthology.orchestra.run.smithi057.stdout:Installed-Size: 19 2022-03-20T14:44:33.159 INFO:teuthology.orchestra.run.smithi057.stdout:Maintainer: Ubuntu Kernel Team 2022-03-20T14:44:33.160 INFO:teuthology.orchestra.run.smithi057.stdout:Architecture: amd64 2022-03-20T14:44:33.160 INFO:teuthology.orchestra.run.smithi057.stdout:Source: linux-meta-hwe-5.13 2022-03-20T14:44:33.160 INFO:teuthology.orchestra.run.smithi057.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-20T14:44:33.160 INFO:teuthology.orchestra.run.smithi057.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-20T14:44:33.160 INFO:teuthology.orchestra.run.smithi057.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-20T14:44:33.161 INFO:teuthology.orchestra.run.smithi057.stdout:Recommends: thermald 2022-03-20T14:44:33.161 INFO:teuthology.orchestra.run.smithi057.stdout:Description: Generic Linux kernel image 2022-03-20T14:44:33.161 INFO:teuthology.orchestra.run.smithi057.stdout: This package will always depend on the latest generic kernel image 2022-03-20T14:44:33.161 INFO:teuthology.orchestra.run.smithi057.stdout: available. 2022-03-20T14:44:33.162 DEBUG:teuthology.orchestra.run.smithi057:> mktemp 2022-03-20T14:44:33.204 INFO:teuthology.orchestra.run.smithi057.stdout:/tmp/tmp.2doe3Sw7P7 2022-03-20T14:44:33.205 DEBUG:teuthology.orchestra.run.smithi057:> sudo cp /boot/grub/grub.cfg /tmp/tmp.2doe3Sw7P7 2022-03-20T14:44:33.219 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0666 /tmp/tmp.2doe3Sw7P7 2022-03-20T14:44:33.259 DEBUG:teuthology.orchestra.remote:smithi057:/tmp/tmp.2doe3Sw7P7 is 8KB 2022-03-20T14:44:33.272 DEBUG:teuthology.orchestra.run.smithi057:> rm -fr /tmp/tmp.2doe3Sw7P7 2022-03-20T14:44:33.275 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2022-03-20T14:44:33.278 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-20T14:44:33.298 DEBUG:teuthology.orchestra.run.smithi007:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-20T14:44:33.334 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2022-03-20T14:44:33.334 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-20T14:44:33.334 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-20T14:44:33.369 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2022-03-20T14:44:33.371 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-35-generic 2022-03-20T14:44:33.371 DEBUG:teuthology.orchestra.run.smithi057:> sudo update-grub 2022-03-20T14:44:33.546 INFO:teuthology.orchestra.run.smithi007.stdout:Building dependency tree... 2022-03-20T14:44:33.547 INFO:teuthology.orchestra.run.smithi007.stdout:Reading state information... 2022-03-20T14:44:33.743 INFO:teuthology.orchestra.run.smithi007.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.35.40~20.04.20). 2022-03-20T14:44:33.743 INFO:teuthology.orchestra.run.smithi007.stdout:0 upgraded, 0 newly installed, 0 to remove and 109 not upgraded. 2022-03-20T14:44:33.745 DEBUG:teuthology.orchestra.run.smithi007:> dpkg -s linux-image-generic-hwe-20.04 2022-03-20T14:44:33.765 INFO:teuthology.orchestra.run.smithi007.stdout:Package: linux-image-generic-hwe-20.04 2022-03-20T14:44:33.765 INFO:teuthology.orchestra.run.smithi007.stdout:Status: install ok installed 2022-03-20T14:44:33.765 INFO:teuthology.orchestra.run.smithi007.stdout:Priority: optional 2022-03-20T14:44:33.765 INFO:teuthology.orchestra.run.smithi007.stdout:Section: kernel 2022-03-20T14:44:33.766 INFO:teuthology.orchestra.run.smithi007.stdout:Installed-Size: 19 2022-03-20T14:44:33.766 INFO:teuthology.orchestra.run.smithi007.stdout:Maintainer: Ubuntu Kernel Team 2022-03-20T14:44:33.766 INFO:teuthology.orchestra.run.smithi007.stdout:Architecture: amd64 2022-03-20T14:44:33.766 INFO:teuthology.orchestra.run.smithi007.stdout:Source: linux-meta-hwe-5.13 2022-03-20T14:44:33.766 INFO:teuthology.orchestra.run.smithi007.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-20T14:44:33.767 INFO:teuthology.orchestra.run.smithi007.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-20T14:44:33.767 INFO:teuthology.orchestra.run.smithi007.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-20T14:44:33.767 INFO:teuthology.orchestra.run.smithi007.stdout:Recommends: thermald 2022-03-20T14:44:33.767 INFO:teuthology.orchestra.run.smithi007.stdout:Description: Generic Linux kernel image 2022-03-20T14:44:33.768 INFO:teuthology.orchestra.run.smithi007.stdout: This package will always depend on the latest generic kernel image 2022-03-20T14:44:33.768 INFO:teuthology.orchestra.run.smithi007.stdout: available. 2022-03-20T14:44:33.768 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2022-03-20T14:44:33.774 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.7sBvB2a5EQ 2022-03-20T14:44:33.774 DEBUG:teuthology.orchestra.run.smithi007:> sudo cp /boot/grub/grub.cfg /tmp/tmp.7sBvB2a5EQ 2022-03-20T14:44:33.790 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.7sBvB2a5EQ 2022-03-20T14:44:33.870 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.7sBvB2a5EQ is 8KB 2022-03-20T14:44:33.884 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.7sBvB2a5EQ 2022-03-20T14:44:33.891 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-20T14:44:33.947 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-20T14:44:33.948 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-20T14:44:33.948 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-20T14:44:33.985 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-35-generic 2022-03-20T14:44:33.986 DEBUG:teuthology.orchestra.run.smithi007:> sudo update-grub 2022-03-20T14:44:34.591 INFO:teuthology.orchestra.run.smithi057.stderr:Sourcing file `/etc/default/grub' 2022-03-20T14:44:34.592 INFO:teuthology.orchestra.run.smithi057.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-20T14:44:34.603 INFO:teuthology.orchestra.run.smithi057.stderr:Generating grub configuration file ... 2022-03-20T14:44:34.838 INFO:teuthology.orchestra.run.smithi057.stderr:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-20T14:44:34.849 INFO:teuthology.orchestra.run.smithi057.stderr:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-20T14:44:35.114 INFO:teuthology.orchestra.run.smithi007.stderr:Sourcing file `/etc/default/grub' 2022-03-20T14:44:35.116 INFO:teuthology.orchestra.run.smithi007.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-20T14:44:35.124 INFO:teuthology.orchestra.run.smithi007.stderr:Generating grub configuration file ... 2022-03-20T14:44:35.199 INFO:teuthology.orchestra.run.smithi057.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-20T14:44:35.211 INFO:teuthology.orchestra.run.smithi057.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-20T14:44:35.414 INFO:teuthology.orchestra.run.smithi007.stderr:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-20T14:44:35.426 INFO:teuthology.orchestra.run.smithi007.stderr:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-20T14:44:35.791 INFO:teuthology.orchestra.run.smithi007.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-20T14:44:35.802 INFO:teuthology.orchestra.run.smithi007.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-20T14:44:36.893 INFO:teuthology.orchestra.run.smithi057.stderr:done 2022-03-20T14:44:36.895 DEBUG:teuthology.orchestra.run.smithi057:> sudo shutdown -r now 2022-03-20T14:44:37.414 INFO:teuthology.orchestra.run.smithi007.stderr:done 2022-03-20T14:44:37.416 DEBUG:teuthology.orchestra.run.smithi007:> sudo shutdown -r now 2022-03-20T14:45:03.034 INFO:teuthology.task.kernel:Checking client host.c for new kernel version... 2022-03-20T14:45:03.035 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2022-03-20T14:45:03.036 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-20T14:45:06.898 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2022-03-20T14:45:06.899 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2022-03-20T14:45:06.900 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-20T14:45:07.419 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2022-03-20T14:45:07.420 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2022-03-20T14:45:07.421 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-20T14:45:21.483 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.87 2022-03-20T14:45:25.323 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.57 2022-03-20T14:45:25.835 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.7 2022-03-20T14:45:51.507 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2022-03-20T14:45:51.508 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-20T14:45:55.327 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2022-03-20T14:45:55.328 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-20T14:45:55.836 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2022-03-20T14:45:55.837 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-20T14:46:09.871 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.87 2022-03-20T14:46:11.747 DEBUG:teuthology.orchestra.run.smithi007:> true 2022-03-20T14:46:12.303 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2022-03-20T14:46:12.303 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.13.0-35-generic"... 2022-03-20T14:46:12.304 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2022-03-20T14:46:12.311 INFO:teuthology.orchestra.run.smithi007.stdout:5.13.0-35-generic 2022-03-20T14:46:12.311 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-35-generic vs 5.13.0-35-generic 2022-03-20T14:46:12.312 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-20T14:46:12.312 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-20T14:46:13.313 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2022-03-20T14:46:13.314 DEBUG:teuthology.orchestra.run.smithi007:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-20T14:46:13.378 INFO:teuthology.orchestra.run.smithi007.stdout:ttyS1 2022-03-20T14:46:13.396 DEBUG:teuthology.parallel:result is None 2022-03-20T14:46:13.706 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.57 2022-03-20T14:46:39.873 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2022-03-20T14:46:39.874 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-20T14:46:40.042 DEBUG:teuthology.orchestra.run.smithi087:> true 2022-03-20T14:46:40.551 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi087.front.sepia.ceph.com' 2022-03-20T14:46:40.551 INFO:teuthology.task.kernel:Checking kernel version of host.c, want "5.13.0-35-generic"... 2022-03-20T14:46:40.552 DEBUG:teuthology.orchestra.run.smithi087:> uname -r 2022-03-20T14:46:40.556 INFO:teuthology.orchestra.run.smithi087.stdout:5.13.0-35-generic 2022-03-20T14:46:40.556 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-35-generic vs 5.13.0-35-generic 2022-03-20T14:46:40.557 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-20T14:46:40.557 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-20T14:46:41.558 INFO:teuthology.task.kernel:Enabling kdb on host.c... 2022-03-20T14:46:41.558 DEBUG:teuthology.orchestra.run.smithi087:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-20T14:46:41.632 INFO:teuthology.orchestra.run.smithi087.stdout:ttyS1 2022-03-20T14:46:41.658 DEBUG:teuthology.parallel:result is None 2022-03-20T14:46:43.707 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2022-03-20T14:46:43.708 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-20T14:46:43.943 DEBUG:teuthology.orchestra.run.smithi057:> true 2022-03-20T14:46:44.459 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi057.front.sepia.ceph.com' 2022-03-20T14:46:44.460 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.13.0-35-generic"... 2022-03-20T14:46:44.460 DEBUG:teuthology.orchestra.run.smithi057:> uname -r 2022-03-20T14:46:44.467 INFO:teuthology.orchestra.run.smithi057.stdout:5.13.0-35-generic 2022-03-20T14:46:44.467 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-35-generic vs 5.13.0-35-generic 2022-03-20T14:46:44.467 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-20T14:46:44.468 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-20T14:46:45.469 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2022-03-20T14:46:45.469 DEBUG:teuthology.orchestra.run.smithi057:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-20T14:46:45.535 INFO:teuthology.orchestra.run.smithi057.stdout:ttyS1 2022-03-20T14:46:45.561 DEBUG:teuthology.parallel:result is None 2022-03-20T14:46:45.561 INFO:teuthology.run_tasks:Running task internal.base... 2022-03-20T14:46:45.570 INFO:teuthology.task.internal:Creating test directory... 2022-03-20T14:46:45.571 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-20T14:46:45.574 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-20T14:46:45.576 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-20T14:46:45.584 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-03-20T14:46:45.610 INFO:teuthology.run_tasks:Running task internal.archive... 2022-03-20T14:46:45.643 INFO:teuthology.task.internal:Creating archive directory... 2022-03-20T14:46:45.643 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-20T14:46:45.646 DEBUG:teuthology.orchestra.run.smithi057:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-20T14:46:45.648 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-20T14:46:45.667 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-03-20T14:46:45.678 INFO:teuthology.task.internal:Enabling coredump saving... 2022-03-20T14:46:45.679 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-20T14:46:45.703 DEBUG:teuthology.orchestra.run.smithi057:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-20T14:46:45.705 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-20T14:46:45.726 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-20T14:46:45.731 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-20T14:46:45.732 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-20T14:46:45.733 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-20T14:46:45.742 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-20T14:46:45.742 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-20T14:46:45.744 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-03-20T14:46:45.755 INFO:teuthology.task.internal:Configuring sudo... 2022-03-20T14:46:45.755 DEBUG:teuthology.orchestra.run.smithi007:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-20T14:46:45.758 DEBUG:teuthology.orchestra.run.smithi057:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-20T14:46:45.760 DEBUG:teuthology.orchestra.run.smithi087:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-20T14:46:45.776 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-03-20T14:46:45.829 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-03-20T14:46:45.830 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-20T14:46:45.832 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-20T14:46:45.835 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-20T14:46:45.842 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-20T14:46:45.888 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-20T14:46:45.894 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-20T14:46:45.894 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-20T14:46:45.928 DEBUG:teuthology.orchestra.run.smithi057:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-20T14:46:45.936 DEBUG:teuthology.orchestra.run.smithi057:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-20T14:46:45.943 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2022-03-20T14:46:45.944 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-20T14:46:46.006 DEBUG:teuthology.orchestra.run.smithi087:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-20T14:46:46.014 DEBUG:teuthology.orchestra.run.smithi087:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-20T14:46:46.021 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2022-03-20T14:46:46.021 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-20T14:46:46.091 DEBUG:teuthology.orchestra.run.smithi007:> sudo service rsyslog restart 2022-03-20T14:46:46.094 DEBUG:teuthology.orchestra.run.smithi057:> sudo service rsyslog restart 2022-03-20T14:46:46.096 DEBUG:teuthology.orchestra.run.smithi087:> sudo service rsyslog restart 2022-03-20T14:46:46.177 INFO:teuthology.run_tasks:Running task internal.timer... 2022-03-20T14:46:46.188 INFO:teuthology.task.internal:Starting timer... 2022-03-20T14:46:46.189 INFO:teuthology.run_tasks:Running task pcp... 2022-03-20T14:46:46.231 INFO:teuthology.run_tasks:Running task selinux... 2022-03-20T14:46:46.262 DEBUG:teuthology.task.selinux:Excluding smithi007: OS 'ubuntu' does not support SELinux 2022-03-20T14:46:46.262 DEBUG:teuthology.task.selinux:Excluding smithi057: OS 'ubuntu' does not support SELinux 2022-03-20T14:46:46.263 DEBUG:teuthology.task.selinux:Excluding smithi087: OS 'ubuntu' does not support SELinux 2022-03-20T14:46:46.263 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-03-20T14:46:46.263 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-03-20T14:46:46.264 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-03-20T14:46:46.264 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-03-20T14:46:46.296 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2022-03-20T14:46:46.303 INFO:teuthology.repo_utils:Fetching master from origin 2022-03-20T14:46:46.338 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2022-03-20T14:46:46.359 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-03-20T14:46:46.361 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi007.front.sepia.ceph.com,smithi057.front.sepia.ceph.com,smithi087.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2022-03-20T14:50:02.406 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi007.front.sepia.ceph.com'), Remote(name='ubuntu@smithi057.front.sepia.ceph.com'), Remote(name='ubuntu@smithi087.front.sepia.ceph.com')] 2022-03-20T14:50:02.408 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2022-03-20T14:50:02.410 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-20T14:50:02.454 DEBUG:teuthology.orchestra.run.smithi007:> true 2022-03-20T14:50:02.576 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2022-03-20T14:50:02.577 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-20T14:50:02.622 DEBUG:teuthology.orchestra.run.smithi057:> true 2022-03-20T14:50:02.748 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2022-03-20T14:50:02.749 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-20T14:50:02.793 DEBUG:teuthology.orchestra.run.smithi087:> true 2022-03-20T14:50:02.916 INFO:teuthology.run_tasks:Running task clock... 2022-03-20T14:50:02.930 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-03-20T14:50:02.931 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-20T14:50:02.933 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-20T14:50:02.935 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-20T14:50:02.936 DEBUG:teuthology.orchestra.run.smithi057:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-20T14:50:02.938 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-20T14:50:02.939 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-20T14:50:02.966 INFO:teuthology.orchestra.run.smithi007.stdout:20 Mar 14:50:02 ntpd[9424]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-20T14:50:02.967 INFO:teuthology.orchestra.run.smithi007.stdout:20 Mar 14:50:02 ntpd[9424]: Command line: ntpd -gq 2022-03-20T14:50:02.967 INFO:teuthology.orchestra.run.smithi007.stdout:20 Mar 14:50:02 ntpd[9424]: proto: precision = 0.090 usec (-23) 2022-03-20T14:50:02.968 INFO:teuthology.orchestra.run.smithi007.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-20T14:50:02.968 INFO:teuthology.orchestra.run.smithi007.stdout:20 Mar 14:50:02 ntpd[9424]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-20T14:50:02.969 INFO:teuthology.orchestra.run.smithi007.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-20T14:50:02.970 INFO:teuthology.orchestra.run.smithi007.stdout:20 Mar 14:50:02 ntpd[9424]: restrict ::: KOD does nothing without LIMITED. 2022-03-20T14:50:02.970 INFO:teuthology.orchestra.run.smithi007.stdout:20 Mar 14:50:02 ntpd[9424]: Listen and drop on 0 v6wildcard [::]:123 2022-03-20T14:50:02.971 INFO:teuthology.orchestra.run.smithi007.stdout:20 Mar 14:50:02 ntpd[9424]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-20T14:50:02.971 INFO:teuthology.orchestra.run.smithi007.stdout:20 Mar 14:50:02 ntpd[9424]: Listen normally on 2 lo 127.0.0.1:123 2022-03-20T14:50:02.971 INFO:teuthology.orchestra.run.smithi007.stdout:20 Mar 14:50:02 ntpd[9424]: Listen normally on 3 ens1f0 172.21.15.7:123 2022-03-20T14:50:02.972 INFO:teuthology.orchestra.run.smithi007.stdout:20 Mar 14:50:02 ntpd[9424]: Listen normally on 4 lo [::1]:123 2022-03-20T14:50:02.972 INFO:teuthology.orchestra.run.smithi007.stdout:20 Mar 14:50:02 ntpd[9424]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:15ea%4]:123 2022-03-20T14:50:02.972 INFO:teuthology.orchestra.run.smithi007.stdout:20 Mar 14:50:02 ntpd[9424]: Listening on routing socket on fd #22 for interface updates 2022-03-20T14:50:02.973 INFO:teuthology.orchestra.run.smithi057.stdout:20 Mar 14:50:02 ntpd[9395]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-20T14:50:02.973 INFO:teuthology.orchestra.run.smithi057.stdout:20 Mar 14:50:02 ntpd[9395]: Command line: ntpd -gq 2022-03-20T14:50:02.974 INFO:teuthology.orchestra.run.smithi057.stdout:20 Mar 14:50:02 ntpd[9395]: proto: precision = 0.072 usec (-24) 2022-03-20T14:50:02.974 INFO:teuthology.orchestra.run.smithi057.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-20T14:50:02.975 INFO:teuthology.orchestra.run.smithi057.stdout:20 Mar 14:50:02 ntpd[9395]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-20T14:50:02.975 INFO:teuthology.orchestra.run.smithi057.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-20T14:50:02.976 INFO:teuthology.orchestra.run.smithi057.stdout:20 Mar 14:50:02 ntpd[9395]: restrict ::: KOD does nothing without LIMITED. 2022-03-20T14:50:02.977 INFO:teuthology.orchestra.run.smithi057.stdout:20 Mar 14:50:02 ntpd[9395]: Listen and drop on 0 v6wildcard [::]:123 2022-03-20T14:50:02.977 INFO:teuthology.orchestra.run.smithi057.stdout:20 Mar 14:50:02 ntpd[9395]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-20T14:50:02.977 INFO:teuthology.orchestra.run.smithi057.stdout:20 Mar 14:50:02 ntpd[9395]: Listen normally on 2 lo 127.0.0.1:123 2022-03-20T14:50:02.978 INFO:teuthology.orchestra.run.smithi087.stdout:20 Mar 14:50:02 ntpd[9518]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-20T14:50:02.979 INFO:teuthology.orchestra.run.smithi087.stdout:20 Mar 14:50:02 ntpd[9518]: Command line: ntpd -gq 2022-03-20T14:50:02.979 INFO:teuthology.orchestra.run.smithi087.stdout:20 Mar 14:50:02 ntpd[9518]: proto: precision = 0.070 usec (-24) 2022-03-20T14:50:02.979 INFO:teuthology.orchestra.run.smithi087.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-20T14:50:02.980 INFO:teuthology.orchestra.run.smithi057.stdout:20 Mar 14:50:02 ntpd[9395]: Listen normally on 3 ens1f0 172.21.15.57:123 2022-03-20T14:50:02.980 INFO:teuthology.orchestra.run.smithi057.stdout:20 Mar 14:50:02 ntpd[9395]: Listen normally on 4 lo [::1]:123 2022-03-20T14:50:02.980 INFO:teuthology.orchestra.run.smithi057.stdout:20 Mar 14:50:02 ntpd[9395]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1562%4]:123 2022-03-20T14:50:02.981 INFO:teuthology.orchestra.run.smithi057.stdout:20 Mar 14:50:02 ntpd[9395]: Listening on routing socket on fd #22 for interface updates 2022-03-20T14:50:02.981 INFO:teuthology.orchestra.run.smithi087.stdout:20 Mar 14:50:02 ntpd[9518]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-20T14:50:02.982 INFO:teuthology.orchestra.run.smithi087.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-20T14:50:02.983 INFO:teuthology.orchestra.run.smithi087.stdout:20 Mar 14:50:02 ntpd[9518]: restrict ::: KOD does nothing without LIMITED. 2022-03-20T14:50:02.984 INFO:teuthology.orchestra.run.smithi087.stdout:20 Mar 14:50:02 ntpd[9518]: Listen and drop on 0 v6wildcard [::]:123 2022-03-20T14:50:02.984 INFO:teuthology.orchestra.run.smithi087.stdout:20 Mar 14:50:02 ntpd[9518]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-20T14:50:02.984 INFO:teuthology.orchestra.run.smithi087.stdout:20 Mar 14:50:02 ntpd[9518]: Listen normally on 2 lo 127.0.0.1:123 2022-03-20T14:50:02.984 INFO:teuthology.orchestra.run.smithi087.stdout:20 Mar 14:50:02 ntpd[9518]: Listen normally on 3 enp3s0f1 172.21.15.87:123 2022-03-20T14:50:02.984 INFO:teuthology.orchestra.run.smithi087.stdout:20 Mar 14:50:02 ntpd[9518]: Listen normally on 4 lo [::1]:123 2022-03-20T14:50:02.985 INFO:teuthology.orchestra.run.smithi087.stdout:20 Mar 14:50:02 ntpd[9518]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d5f5%5]:123 2022-03-20T14:50:02.985 INFO:teuthology.orchestra.run.smithi087.stdout:20 Mar 14:50:02 ntpd[9518]: Listening on routing socket on fd #22 for interface updates 2022-03-20T14:50:03.967 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:03 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:03.968 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:03 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:03.968 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:03 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:03.968 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:03 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:03.968 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:03 ntpd[9424]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:50:03.968 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:03 ntpd[9424]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:50:03.974 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:03 ntpd[9518]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:03.974 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:03 ntpd[9518]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:03.975 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:03 ntpd[9518]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:03.975 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:03 ntpd[9518]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:03.975 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:03 ntpd[9518]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:50:03.975 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:03 ntpd[9518]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:50:04.967 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:04 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:04.967 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:04 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:04.968 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:04 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:04.968 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:04 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:04.968 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:04 ntpd[9424]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:50:04.968 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:04 ntpd[9424]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:50:04.970 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:04 ntpd[9395]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:04.970 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:04 ntpd[9395]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:04.971 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:04 ntpd[9395]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:04.971 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:04 ntpd[9395]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:04.971 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:04 ntpd[9395]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:04.971 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:04 ntpd[9395]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:04.972 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:04 ntpd[9395]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:50:04.972 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:04 ntpd[9395]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:50:04.972 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:04 ntpd[9395]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:04.972 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:04 ntpd[9395]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:04.972 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:04 ntpd[9395]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:50:04.973 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:04 ntpd[9395]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:50:05.968 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:05 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:05.969 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:05 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:05.969 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:05 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:05.970 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:05 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:05.970 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:05 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:05.970 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:05 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:05.970 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:05 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:05.970 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:05 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:05.971 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:05 ntpd[9424]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:50:05.971 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:05 ntpd[9424]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:50:05.971 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:05 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:05.971 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:05 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:05.972 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:05 ntpd[9424]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:50:05.972 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:05 ntpd[9424]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:50:05.972 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:05 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:05.972 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:05 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:05.973 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:05 ntpd[9424]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:50:05.973 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:05 ntpd[9424]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:50:05.974 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:05 ntpd[9518]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:05.974 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:05 ntpd[9518]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:05.975 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:05 ntpd[9518]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:05.975 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:05 ntpd[9518]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:05.975 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:05 ntpd[9518]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:05.975 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:05 ntpd[9518]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:05.976 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:05 ntpd[9518]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:50:05.976 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:05 ntpd[9518]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:50:05.976 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:05 ntpd[9518]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:05.976 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:05 ntpd[9518]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:05.976 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:05 ntpd[9518]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:50:05.977 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:05 ntpd[9518]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:50:06.967 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:06 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:06.968 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:06 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:06.968 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:06 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:06.968 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:06 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:06.968 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:06 ntpd[9424]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:50:06.969 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:06 ntpd[9424]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:50:06.970 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:06 ntpd[9395]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:06.971 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:06 ntpd[9395]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:06.971 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:06 ntpd[9395]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:06.971 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:06 ntpd[9395]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:06.971 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:06 ntpd[9395]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:06.971 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:06 ntpd[9395]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:06.972 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:06 ntpd[9395]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:50:06.972 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:06 ntpd[9395]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:50:06.972 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:06 ntpd[9395]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:06.972 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:06 ntpd[9395]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:06.973 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:06 ntpd[9395]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:50:06.973 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:06 ntpd[9395]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:50:06.975 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:06 ntpd[9518]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:06.975 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:06 ntpd[9518]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:06.975 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:06 ntpd[9518]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:06.976 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:06 ntpd[9518]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:06.976 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:06 ntpd[9518]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:06.976 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:06 ntpd[9518]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:06.976 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:06 ntpd[9518]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:50:06.976 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:06 ntpd[9518]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:50:06.977 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:06 ntpd[9518]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:06.977 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:06 ntpd[9518]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:06.977 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:06 ntpd[9518]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:50:06.977 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:06 ntpd[9518]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:50:07.968 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:07 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:07.969 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:07 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:07.969 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:07 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:07.969 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:07 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:07.969 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:07 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:07.970 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:07 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:07.970 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:07 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:07.970 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:07 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:07.970 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:07 ntpd[9424]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:50:07.970 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:07 ntpd[9424]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:50:07.971 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:07 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:07.971 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:07 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:07.971 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:07 ntpd[9424]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:50:07.971 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:07 ntpd[9424]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:50:07.972 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:07 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:07.972 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:07 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:07.972 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:07 ntpd[9424]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:50:07.972 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:07 ntpd[9424]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:50:07.974 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:07 ntpd[9395]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:07.974 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:07 ntpd[9395]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:07.975 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:07 ntpd[9395]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:07.975 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:07 ntpd[9395]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:07.975 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:07 ntpd[9395]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:07.975 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:07 ntpd[9395]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:07.975 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:07 ntpd[9395]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:50:07.976 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:07 ntpd[9395]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:50:07.976 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:07 ntpd[9395]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:07.976 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:07 ntpd[9395]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:07.976 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:07 ntpd[9395]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:50:07.977 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:07 ntpd[9395]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:50:07.978 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:07 ntpd[9518]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:07.978 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:07 ntpd[9518]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:07.979 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:07 ntpd[9518]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:07.979 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:07 ntpd[9518]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:07.979 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:07 ntpd[9518]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:07.979 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:07 ntpd[9518]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:07.980 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:07 ntpd[9518]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:50:07.980 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:07 ntpd[9518]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:50:07.980 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:07 ntpd[9518]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:07.980 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:07 ntpd[9518]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:07.981 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:07 ntpd[9518]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:50:07.981 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:07 ntpd[9518]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:50:08.967 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:08 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:08.968 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:08 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:08.968 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:08 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:08.968 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:08 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:08.968 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:08 ntpd[9424]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:50:08.968 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:08 ntpd[9424]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:50:08.970 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:08 ntpd[9395]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:08.970 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:08 ntpd[9395]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:08.971 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:08 ntpd[9395]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:08.971 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:08 ntpd[9395]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:08.971 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:08 ntpd[9395]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:08.971 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:08 ntpd[9395]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:08.972 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:08 ntpd[9395]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:50:08.972 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:08 ntpd[9395]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:50:08.972 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:08 ntpd[9395]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:08.972 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:08 ntpd[9395]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:08.973 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:08 ntpd[9395]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:50:08.973 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:08 ntpd[9395]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:50:08.975 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:08 ntpd[9518]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:08.975 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:08 ntpd[9518]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:08.975 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:08 ntpd[9518]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:08.975 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:08 ntpd[9518]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:08.976 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:08 ntpd[9518]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:08.976 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:08 ntpd[9518]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:08.976 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:08 ntpd[9518]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:50:08.976 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:08 ntpd[9518]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:50:08.976 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:08 ntpd[9518]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:08.977 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:08 ntpd[9518]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:08.977 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:08 ntpd[9518]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:50:08.977 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:08 ntpd[9518]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:50:09.968 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:09 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:09.968 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:09 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:09.969 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:09 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:09.969 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:09 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:09.969 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:09 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:09.969 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:09 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:09.970 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:09 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:09.970 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:09 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:09.970 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:09 ntpd[9424]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:50:09.970 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:09 ntpd[9424]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:50:09.970 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:09 ntpd[9424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:09.971 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:09 ntpd[9424]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:09.971 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:09 ntpd[9424]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:50:09.971 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:09 ntpd[9424]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:50:09.972 INFO:teuthology.orchestra.run.smithi007.stdout:20 Mar 14:50:09 ntpd[9424]: ntpd: time slew +0.000409 s 2022-03-20T14:50:09.972 INFO:teuthology.orchestra.run.smithi007.stdout:ntpd: time slew +0.000409s 2022-03-20T14:50:09.972 INFO:teuthology.orchestra.run.smithi007.stderr:20 Mar 14:50:09 ntpd[9424]: can't open /var/log/ntpstats/loopstats.20220320: Permission denied 2022-03-20T14:50:09.974 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:09 ntpd[9395]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:09.975 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:09 ntpd[9395]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:09.975 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:09 ntpd[9395]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:09.975 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:09 ntpd[9395]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:09.975 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:09 ntpd[9395]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:09.976 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:09 ntpd[9395]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:09.976 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:09 ntpd[9395]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:50:09.976 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:09 ntpd[9395]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:50:09.977 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:09 ntpd[9395]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:09.977 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:09 ntpd[9395]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:09.977 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:09 ntpd[9395]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:50:09.977 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:09 ntpd[9395]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:50:09.978 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:09 ntpd[9518]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:09.979 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:09 ntpd[9518]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:09.979 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:09 ntpd[9518]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:09.979 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:09 ntpd[9518]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:09.979 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:09 ntpd[9518]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:09.980 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:09 ntpd[9518]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:09.980 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:09 ntpd[9518]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:50:09.980 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:09 ntpd[9518]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:50:09.981 INFO:teuthology.orchestra.run.smithi087.stdout:20 Mar 14:50:09 ntpd[9518]: ntpd: time slew -0.002158 s 2022-03-20T14:50:09.981 INFO:teuthology.orchestra.run.smithi087.stdout:ntpd: time slew -0.002158s 2022-03-20T14:50:09.981 INFO:teuthology.orchestra.run.smithi087.stderr:20 Mar 14:50:09 ntpd[9518]: can't open /var/log/ntpstats/loopstats.20220320: Permission denied 2022-03-20T14:50:10.031 INFO:teuthology.orchestra.run.smithi007.stdout: remote refid st t when poll reach delay offset jitter 2022-03-20T14:50:10.031 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================== 2022-03-20T14:50:10.031 INFO:teuthology.orchestra.run.smithi007.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-20T14:50:10.031 INFO:teuthology.orchestra.run.smithi007.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-20T14:50:10.032 INFO:teuthology.orchestra.run.smithi007.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-20T14:50:10.032 INFO:teuthology.orchestra.run.smithi007.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-20T14:50:10.047 INFO:teuthology.orchestra.run.smithi087.stdout: remote refid st t when poll reach delay offset jitter 2022-03-20T14:50:10.047 INFO:teuthology.orchestra.run.smithi087.stdout:============================================================================== 2022-03-20T14:50:10.047 INFO:teuthology.orchestra.run.smithi087.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-20T14:50:10.047 INFO:teuthology.orchestra.run.smithi087.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-20T14:50:10.048 INFO:teuthology.orchestra.run.smithi087.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-20T14:50:10.048 INFO:teuthology.orchestra.run.smithi087.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-20T14:50:10.970 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:10 ntpd[9395]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:10.971 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:10 ntpd[9395]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:10.971 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:10 ntpd[9395]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:10.971 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:10 ntpd[9395]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:10.971 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:10 ntpd[9395]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-20T14:50:10.972 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:10 ntpd[9395]: can't open /var/log/ntpstats/rawstats.20220320: Permission denied 2022-03-20T14:50:10.972 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:10 ntpd[9395]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-20T14:50:10.972 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:10 ntpd[9395]: can't open /var/log/ntpstats/peerstats.20220320: Permission denied 2022-03-20T14:50:10.973 INFO:teuthology.orchestra.run.smithi057.stdout:20 Mar 14:50:10 ntpd[9395]: ntpd: time slew -0.001716 s 2022-03-20T14:50:10.973 INFO:teuthology.orchestra.run.smithi057.stdout:ntpd: time slew -0.001716s 2022-03-20T14:50:10.973 INFO:teuthology.orchestra.run.smithi057.stderr:20 Mar 14:50:10 ntpd[9395]: can't open /var/log/ntpstats/loopstats.20220320: Permission denied 2022-03-20T14:50:11.030 INFO:teuthology.orchestra.run.smithi057.stdout: remote refid st t when poll reach delay offset jitter 2022-03-20T14:50:11.031 INFO:teuthology.orchestra.run.smithi057.stdout:============================================================================== 2022-03-20T14:50:11.031 INFO:teuthology.orchestra.run.smithi057.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-20T14:50:11.031 INFO:teuthology.orchestra.run.smithi057.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-20T14:50:11.031 INFO:teuthology.orchestra.run.smithi057.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-20T14:50:11.032 INFO:teuthology.orchestra.run.smithi057.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-20T14:50:11.033 INFO:teuthology.run_tasks:Running task kubeadm... 2022-03-20T14:50:11.054 INFO:tasks.kubeadm:Kubeadm start 2022-03-20T14:50:11.054 INFO:tasks.kubeadm:Config: {'pod_network': 'flannel', 'version': '1.21'} 2022-03-20T14:50:11.055 DEBUG:teuthology.orchestra.run.smithi007:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-20T14:50:11.058 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-20T14:50:11.077 DEBUG:teuthology.orchestra.run.smithi087:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-20T14:50:11.079 INFO:teuthology.orchestra.run.smithi007.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-20T14:50:11.085 INFO:teuthology.orchestra.run.smithi007.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-20T14:50:11.093 INFO:teuthology.orchestra.run.smithi007.stdout:net.ipv4.ip_forward = 1 2022-03-20T14:50:11.100 INFO:teuthology.orchestra.run.smithi057.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-20T14:50:11.103 INFO:teuthology.orchestra.run.smithi087.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-20T14:50:11.108 INFO:teuthology.orchestra.run.smithi057.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-20T14:50:11.113 INFO:teuthology.orchestra.run.smithi087.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-20T14:50:11.116 INFO:teuthology.orchestra.run.smithi057.stdout:net.ipv4.ip_forward = 1 2022-03-20T14:50:11.124 INFO:teuthology.orchestra.run.smithi087.stdout:net.ipv4.ip_forward = 1 2022-03-20T14:50:11.170 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-20T14:50:11.171 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-20T14:50:11.187 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-20T14:50:11.187 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/docker/daemon.json 2022-03-20T14:50:11.244 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2022-03-20T14:50:11.245 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-20T14:50:11.260 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2022-03-20T14:50:11.261 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/docker/daemon.json 2022-03-20T14:50:11.274 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2022-03-20T14:50:11.275 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-20T14:50:11.291 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2022-03-20T14:50:11.292 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/docker/daemon.json 2022-03-20T14:50:11.306 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl restart docker || true 2022-03-20T14:50:11.308 DEBUG:teuthology.orchestra.run.smithi057:> sudo systemctl restart docker || true 2022-03-20T14:50:11.317 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl restart docker || true 2022-03-20T14:50:12.331 INFO:tasks.kubeadm:Identifying pod subnet 2022-03-20T14:50:12.339 INFO:tasks.kubeadm:172.21.15.7 is in 172.21.0.0/20 at pos 3846 2022-03-20T14:50:12.352 INFO:tasks.kubeadm:Pod subnet: 10.248.48.0/21 2022-03-20T14:50:12.353 INFO:tasks.kubeadm:Installing kube{adm,ctl,let} 2022-03-20T14:50:12.353 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-20T14:50:12.356 DEBUG:teuthology.orchestra.run.smithi057:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-20T14:50:12.359 DEBUG:teuthology.orchestra.run.smithi087:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-20T14:50:12.380 INFO:teuthology.orchestra.run.smithi007.stderr: 2022-03-20T14:50:12.380 INFO:teuthology.orchestra.run.smithi007.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-20T14:50:12.381 INFO:teuthology.orchestra.run.smithi007.stderr: 2022-03-20T14:50:12.382 INFO:teuthology.orchestra.run.smithi057.stderr: 2022-03-20T14:50:12.383 INFO:teuthology.orchestra.run.smithi057.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-20T14:50:12.383 INFO:teuthology.orchestra.run.smithi057.stderr: 2022-03-20T14:50:12.390 INFO:teuthology.orchestra.run.smithi087.stderr: 2022-03-20T14:50:12.390 INFO:teuthology.orchestra.run.smithi087.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-20T14:50:12.391 INFO:teuthology.orchestra.run.smithi087.stderr: 2022-03-20T14:50:12.495 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-20T14:50:12.501 INFO:teuthology.orchestra.run.smithi057.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-20T14:50:12.503 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-20T14:50:12.504 INFO:teuthology.orchestra.run.smithi087.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-20T14:50:12.523 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-20T14:50:12.603 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-20T14:50:12.609 INFO:teuthology.orchestra.run.smithi057.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-20T14:50:12.612 INFO:teuthology.orchestra.run.smithi087.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-20T14:50:12.691 INFO:teuthology.orchestra.run.smithi057.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-20T14:50:12.692 INFO:teuthology.orchestra.run.smithi087.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-20T14:50:12.771 INFO:teuthology.orchestra.run.smithi057.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-20T14:50:12.773 INFO:teuthology.orchestra.run.smithi087.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-20T14:50:14.004 INFO:teuthology.orchestra.run.smithi057.stdout:Reading package lists... 2022-03-20T14:50:14.007 INFO:teuthology.orchestra.run.smithi087.stdout:Reading package lists... 2022-03-20T14:50:14.033 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2022-03-20T14:50:14.176 INFO:teuthology.orchestra.run.smithi087.stdout:Building dependency tree... 2022-03-20T14:50:14.177 INFO:teuthology.orchestra.run.smithi087.stdout:Reading state information... 2022-03-20T14:50:14.180 INFO:teuthology.orchestra.run.smithi057.stdout:Building dependency tree... 2022-03-20T14:50:14.181 INFO:teuthology.orchestra.run.smithi057.stdout:Reading state information... 2022-03-20T14:50:14.210 INFO:teuthology.orchestra.run.smithi007.stdout:Building dependency tree... 2022-03-20T14:50:14.211 INFO:teuthology.orchestra.run.smithi007.stdout:Reading state information... 2022-03-20T14:50:14.270 INFO:teuthology.orchestra.run.smithi087.stdout:98 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-20T14:50:14.275 INFO:teuthology.orchestra.run.smithi057.stdout:98 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-20T14:50:14.287 INFO:teuthology.orchestra.run.smithi087.stderr: 2022-03-20T14:50:14.287 INFO:teuthology.orchestra.run.smithi087.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-20T14:50:14.287 INFO:teuthology.orchestra.run.smithi087.stderr: 2022-03-20T14:50:14.292 INFO:teuthology.orchestra.run.smithi057.stderr: 2022-03-20T14:50:14.292 INFO:teuthology.orchestra.run.smithi057.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-20T14:50:14.292 INFO:teuthology.orchestra.run.smithi057.stderr: 2022-03-20T14:50:14.305 INFO:teuthology.orchestra.run.smithi007.stdout:98 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-20T14:50:14.322 INFO:teuthology.orchestra.run.smithi007.stderr: 2022-03-20T14:50:14.323 INFO:teuthology.orchestra.run.smithi007.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-20T14:50:14.323 INFO:teuthology.orchestra.run.smithi007.stderr: 2022-03-20T14:50:14.341 INFO:teuthology.orchestra.run.smithi087.stdout:Reading package lists... 2022-03-20T14:50:14.347 INFO:teuthology.orchestra.run.smithi057.stdout:Reading package lists... 2022-03-20T14:50:14.374 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2022-03-20T14:50:14.516 INFO:teuthology.orchestra.run.smithi087.stdout:Building dependency tree... 2022-03-20T14:50:14.517 INFO:teuthology.orchestra.run.smithi087.stdout:Reading state information... 2022-03-20T14:50:14.529 INFO:teuthology.orchestra.run.smithi057.stdout:Building dependency tree... 2022-03-20T14:50:14.530 INFO:teuthology.orchestra.run.smithi057.stdout:Reading state information... 2022-03-20T14:50:14.536 INFO:teuthology.orchestra.run.smithi007.stdout:Building dependency tree... 2022-03-20T14:50:14.537 INFO:teuthology.orchestra.run.smithi007.stdout:Reading state information... 2022-03-20T14:50:14.697 INFO:teuthology.orchestra.run.smithi087.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-20T14:50:14.697 INFO:teuthology.orchestra.run.smithi087.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-20T14:50:14.697 INFO:teuthology.orchestra.run.smithi087.stdout:The following NEW packages will be installed: 2022-03-20T14:50:14.698 INFO:teuthology.orchestra.run.smithi087.stdout: apt-transport-https 2022-03-20T14:50:14.710 INFO:teuthology.orchestra.run.smithi007.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-20T14:50:14.710 INFO:teuthology.orchestra.run.smithi007.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-20T14:50:14.711 INFO:teuthology.orchestra.run.smithi007.stdout:The following NEW packages will be installed: 2022-03-20T14:50:14.711 INFO:teuthology.orchestra.run.smithi007.stdout: apt-transport-https 2022-03-20T14:50:14.719 INFO:teuthology.orchestra.run.smithi057.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-20T14:50:14.720 INFO:teuthology.orchestra.run.smithi057.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-20T14:50:14.720 INFO:teuthology.orchestra.run.smithi057.stdout:The following NEW packages will be installed: 2022-03-20T14:50:14.721 INFO:teuthology.orchestra.run.smithi057.stdout: apt-transport-https 2022-03-20T14:50:14.760 INFO:teuthology.orchestra.run.smithi087.stdout:0 upgraded, 1 newly installed, 0 to remove and 98 not upgraded. 2022-03-20T14:50:14.760 INFO:teuthology.orchestra.run.smithi087.stdout:Need to get 4,680 B of archives. 2022-03-20T14:50:14.761 INFO:teuthology.orchestra.run.smithi087.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-20T14:50:14.761 INFO:teuthology.orchestra.run.smithi087.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-20T14:50:14.782 INFO:teuthology.orchestra.run.smithi057.stdout:0 upgraded, 1 newly installed, 0 to remove and 98 not upgraded. 2022-03-20T14:50:14.782 INFO:teuthology.orchestra.run.smithi057.stdout:Need to get 4,680 B of archives. 2022-03-20T14:50:14.782 INFO:teuthology.orchestra.run.smithi057.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-20T14:50:14.782 INFO:teuthology.orchestra.run.smithi057.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-20T14:50:14.891 INFO:teuthology.orchestra.run.smithi007.stdout:0 upgraded, 1 newly installed, 0 to remove and 98 not upgraded. 2022-03-20T14:50:14.891 INFO:teuthology.orchestra.run.smithi007.stdout:Need to get 4,680 B of archives. 2022-03-20T14:50:14.891 INFO:teuthology.orchestra.run.smithi007.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-20T14:50:14.891 INFO:teuthology.orchestra.run.smithi007.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-20T14:50:14.991 INFO:teuthology.orchestra.run.smithi087.stderr:debconf: unable to initialize frontend: Dialog 2022-03-20T14:50:14.991 INFO:teuthology.orchestra.run.smithi087.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-20T14:50:14.992 INFO:teuthology.orchestra.run.smithi087.stderr:debconf: falling back to frontend: Readline 2022-03-20T14:50:15.009 INFO:teuthology.orchestra.run.smithi087.stderr:debconf: unable to initialize frontend: Readline 2022-03-20T14:50:15.009 INFO:teuthology.orchestra.run.smithi087.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-20T14:50:15.009 INFO:teuthology.orchestra.run.smithi087.stderr:debconf: falling back to frontend: Teletype 2022-03-20T14:50:15.015 INFO:teuthology.orchestra.run.smithi087.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-20T14:50:15.018 INFO:teuthology.orchestra.run.smithi057.stderr:debconf: unable to initialize frontend: Dialog 2022-03-20T14:50:15.018 INFO:teuthology.orchestra.run.smithi057.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-20T14:50:15.019 INFO:teuthology.orchestra.run.smithi057.stderr:debconf: falling back to frontend: Readline 2022-03-20T14:50:15.048 INFO:teuthology.orchestra.run.smithi057.stderr:debconf: unable to initialize frontend: Readline 2022-03-20T14:50:15.049 INFO:teuthology.orchestra.run.smithi057.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-20T14:50:15.049 INFO:teuthology.orchestra.run.smithi057.stderr:debconf: falling back to frontend: Teletype 2022-03-20T14:50:15.054 INFO:teuthology.orchestra.run.smithi057.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-20T14:50:15.060 INFO:teuthology.orchestra.run.smithi087.stdout:Fetched 4,680 B in 0s (88.0 kB/s) 2022-03-20T14:50:15.106 INFO:teuthology.orchestra.run.smithi057.stdout:Fetched 4,680 B in 0s (89.4 kB/s) 2022-03-20T14:50:15.129 INFO:teuthology.orchestra.run.smithi007.stderr:debconf: unable to initialize frontend: Dialog 2022-03-20T14:50:15.129 INFO:teuthology.orchestra.run.smithi007.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-20T14:50:15.130 INFO:teuthology.orchestra.run.smithi007.stderr:debconf: falling back to frontend: Readline 2022-03-20T14:50:15.145 INFO:teuthology.orchestra.run.smithi007.stderr:debconf: unable to initialize frontend: Readline 2022-03-20T14:50:15.145 INFO:teuthology.orchestra.run.smithi007.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-20T14:50:15.146 INFO:teuthology.orchestra.run.smithi007.stderr:debconf: falling back to frontend: Teletype 2022-03-20T14:50:15.151 INFO:teuthology.orchestra.run.smithi007.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-20T14:50:15.157 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package apt-transport-https. 2022-03-20T14:50:15.199 INFO:teuthology.orchestra.run.smithi087.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-20T14:50:15.200 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package apt-transport-https. 2022-03-20T14:50:15.203 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-20T14:50:15.204 INFO:teuthology.orchestra.run.smithi007.stdout:Fetched 4,680 B in 0s (27.2 kB/s) 2022-03-20T14:50:15.246 INFO:teuthology.orchestra.run.smithi057.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-20T14:50:15.247 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-20T14:50:15.249 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-20T14:50:15.290 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-20T14:50:15.294 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package apt-transport-https. 2022-03-20T14:50:15.336 INFO:teuthology.orchestra.run.smithi007.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-20T14:50:15.339 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-20T14:50:15.384 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-20T14:50:15.511 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-20T14:50:15.553 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-20T14:50:15.640 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-20T14:50:16.070 INFO:teuthology.orchestra.run.smithi087.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-20T14:50:16.077 INFO:teuthology.orchestra.run.smithi057.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-20T14:50:16.083 INFO:teuthology.orchestra.run.smithi087.stderr: 2022-03-20T14:50:16.084 INFO:teuthology.orchestra.run.smithi087.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-20T14:50:16.084 INFO:teuthology.orchestra.run.smithi087.stderr: 2022-03-20T14:50:16.093 INFO:teuthology.orchestra.run.smithi057.stderr: 2022-03-20T14:50:16.094 INFO:teuthology.orchestra.run.smithi057.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-20T14:50:16.094 INFO:teuthology.orchestra.run.smithi057.stderr: 2022-03-20T14:50:16.215 INFO:teuthology.orchestra.run.smithi087.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-20T14:50:16.223 INFO:teuthology.orchestra.run.smithi087.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-20T14:50:16.227 INFO:teuthology.orchestra.run.smithi057.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-20T14:50:16.232 INFO:teuthology.orchestra.run.smithi007.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-20T14:50:16.235 INFO:teuthology.orchestra.run.smithi057.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-20T14:50:16.243 INFO:teuthology.orchestra.run.smithi087.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-20T14:50:16.246 INFO:teuthology.orchestra.run.smithi007.stderr: 2022-03-20T14:50:16.246 INFO:teuthology.orchestra.run.smithi007.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-20T14:50:16.247 INFO:teuthology.orchestra.run.smithi007.stderr: 2022-03-20T14:50:16.255 INFO:teuthology.orchestra.run.smithi057.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-20T14:50:16.323 INFO:teuthology.orchestra.run.smithi087.stdout:Hit:5 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-20T14:50:16.338 INFO:teuthology.orchestra.run.smithi057.stdout:Hit:5 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-20T14:50:16.499 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-20T14:50:16.500 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:3 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-20T14:50:16.567 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-20T14:50:16.623 INFO:teuthology.orchestra.run.smithi087.stdout:Get:4 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-20T14:50:16.648 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-20T14:50:16.827 INFO:teuthology.orchestra.run.smithi057.stdout:Get:4 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-20T14:50:16.849 INFO:teuthology.orchestra.run.smithi087.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [54.7 kB] 2022-03-20T14:50:16.941 INFO:teuthology.orchestra.run.smithi057.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [54.7 kB] 2022-03-20T14:50:16.982 INFO:teuthology.orchestra.run.smithi087.stdout:Fetched 64.1 kB in 1s (85.1 kB/s) 2022-03-20T14:50:17.076 INFO:teuthology.orchestra.run.smithi057.stdout:Fetched 64.1 kB in 1s (77.0 kB/s) 2022-03-20T14:50:17.150 INFO:teuthology.orchestra.run.smithi007.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-20T14:50:17.261 INFO:teuthology.orchestra.run.smithi007.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [54.7 kB] 2022-03-20T14:50:17.391 INFO:teuthology.orchestra.run.smithi007.stdout:Fetched 64.1 kB in 1s (64.3 kB/s) 2022-03-20T14:50:17.835 INFO:teuthology.orchestra.run.smithi087.stdout:Reading package lists... 2022-03-20T14:50:17.965 INFO:teuthology.orchestra.run.smithi057.stdout:Reading package lists... 2022-03-20T14:50:18.007 INFO:teuthology.orchestra.run.smithi087.stdout:Building dependency tree... 2022-03-20T14:50:18.008 INFO:teuthology.orchestra.run.smithi087.stdout:Reading state information... 2022-03-20T14:50:18.102 INFO:teuthology.orchestra.run.smithi087.stdout:98 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-20T14:50:18.114 INFO:teuthology.orchestra.run.smithi087.stderr: 2022-03-20T14:50:18.114 INFO:teuthology.orchestra.run.smithi087.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-20T14:50:18.114 INFO:teuthology.orchestra.run.smithi087.stderr: 2022-03-20T14:50:18.143 INFO:teuthology.orchestra.run.smithi057.stdout:Building dependency tree... 2022-03-20T14:50:18.143 INFO:teuthology.orchestra.run.smithi057.stdout:Reading state information... 2022-03-20T14:50:18.167 INFO:teuthology.orchestra.run.smithi087.stdout:Reading package lists... 2022-03-20T14:50:18.240 INFO:teuthology.orchestra.run.smithi057.stdout:98 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-20T14:50:18.253 INFO:teuthology.orchestra.run.smithi057.stderr: 2022-03-20T14:50:18.254 INFO:teuthology.orchestra.run.smithi057.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-20T14:50:18.254 INFO:teuthology.orchestra.run.smithi057.stderr: 2022-03-20T14:50:18.272 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2022-03-20T14:50:18.310 INFO:teuthology.orchestra.run.smithi057.stdout:Reading package lists... 2022-03-20T14:50:18.327 INFO:teuthology.orchestra.run.smithi087.stdout:Building dependency tree... 2022-03-20T14:50:18.328 INFO:teuthology.orchestra.run.smithi087.stdout:Reading state information... 2022-03-20T14:50:18.449 INFO:teuthology.orchestra.run.smithi007.stdout:Building dependency tree... 2022-03-20T14:50:18.450 INFO:teuthology.orchestra.run.smithi007.stdout:Reading state information... 2022-03-20T14:50:18.463 INFO:teuthology.orchestra.run.smithi087.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-20T14:50:18.464 INFO:teuthology.orchestra.run.smithi087.stdout:bridge-utils set to manually installed. 2022-03-20T14:50:18.464 INFO:teuthology.orchestra.run.smithi087.stdout:The following additional packages will be installed: 2022-03-20T14:50:18.465 INFO:teuthology.orchestra.run.smithi087.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-20T14:50:18.465 INFO:teuthology.orchestra.run.smithi087.stdout:Suggested packages: 2022-03-20T14:50:18.465 INFO:teuthology.orchestra.run.smithi087.stdout: nftables 2022-03-20T14:50:18.487 INFO:teuthology.orchestra.run.smithi057.stdout:Building dependency tree... 2022-03-20T14:50:18.488 INFO:teuthology.orchestra.run.smithi057.stdout:Reading state information... 2022-03-20T14:50:18.512 INFO:teuthology.orchestra.run.smithi087.stdout:The following NEW packages will be installed: 2022-03-20T14:50:18.513 INFO:teuthology.orchestra.run.smithi087.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-20T14:50:18.546 INFO:teuthology.orchestra.run.smithi007.stdout:98 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-20T14:50:18.562 INFO:teuthology.orchestra.run.smithi007.stderr: 2022-03-20T14:50:18.562 INFO:teuthology.orchestra.run.smithi007.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-20T14:50:18.562 INFO:teuthology.orchestra.run.smithi007.stderr: 2022-03-20T14:50:18.590 INFO:teuthology.orchestra.run.smithi087.stdout:0 upgraded, 8 newly installed, 0 to remove and 98 not upgraded. 2022-03-20T14:50:18.590 INFO:teuthology.orchestra.run.smithi087.stdout:Need to get 77.7 MB of archives. 2022-03-20T14:50:18.590 INFO:teuthology.orchestra.run.smithi087.stdout:After this operation, 335 MB of additional disk space will be used. 2022-03-20T14:50:18.590 INFO:teuthology.orchestra.run.smithi087.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-20T14:50:18.620 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2022-03-20T14:50:18.630 INFO:teuthology.orchestra.run.smithi057.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-20T14:50:18.630 INFO:teuthology.orchestra.run.smithi057.stdout:bridge-utils set to manually installed. 2022-03-20T14:50:18.631 INFO:teuthology.orchestra.run.smithi057.stdout:The following additional packages will be installed: 2022-03-20T14:50:18.631 INFO:teuthology.orchestra.run.smithi057.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-20T14:50:18.632 INFO:teuthology.orchestra.run.smithi057.stdout:Suggested packages: 2022-03-20T14:50:18.632 INFO:teuthology.orchestra.run.smithi057.stdout: nftables 2022-03-20T14:50:18.641 INFO:teuthology.orchestra.run.smithi087.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-20T14:50:18.676 INFO:teuthology.orchestra.run.smithi087.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-20T14:50:18.680 INFO:teuthology.orchestra.run.smithi057.stdout:The following NEW packages will be installed: 2022-03-20T14:50:18.681 INFO:teuthology.orchestra.run.smithi057.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-20T14:50:18.770 INFO:teuthology.orchestra.run.smithi057.stdout:0 upgraded, 8 newly installed, 0 to remove and 98 not upgraded. 2022-03-20T14:50:18.770 INFO:teuthology.orchestra.run.smithi057.stdout:Need to get 77.7 MB of archives. 2022-03-20T14:50:18.771 INFO:teuthology.orchestra.run.smithi057.stdout:After this operation, 335 MB of additional disk space will be used. 2022-03-20T14:50:18.771 INFO:teuthology.orchestra.run.smithi057.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-20T14:50:18.799 INFO:teuthology.orchestra.run.smithi007.stdout:Building dependency tree... 2022-03-20T14:50:18.799 INFO:teuthology.orchestra.run.smithi007.stdout:Reading state information... 2022-03-20T14:50:18.834 INFO:teuthology.orchestra.run.smithi087.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.23.0-00 [15.3 MB] 2022-03-20T14:50:18.837 INFO:teuthology.orchestra.run.smithi057.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-20T14:50:18.880 INFO:teuthology.orchestra.run.smithi057.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-20T14:50:18.942 INFO:teuthology.orchestra.run.smithi007.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-20T14:50:18.943 INFO:teuthology.orchestra.run.smithi007.stdout:bridge-utils set to manually installed. 2022-03-20T14:50:18.943 INFO:teuthology.orchestra.run.smithi007.stdout:The following additional packages will be installed: 2022-03-20T14:50:18.944 INFO:teuthology.orchestra.run.smithi007.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-20T14:50:18.944 INFO:teuthology.orchestra.run.smithi007.stdout:Suggested packages: 2022-03-20T14:50:18.944 INFO:teuthology.orchestra.run.smithi007.stdout: nftables 2022-03-20T14:50:18.992 INFO:teuthology.orchestra.run.smithi007.stdout:The following NEW packages will be installed: 2022-03-20T14:50:18.993 INFO:teuthology.orchestra.run.smithi007.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-20T14:50:19.047 INFO:teuthology.orchestra.run.smithi057.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.23.0-00 [15.3 MB] 2022-03-20T14:50:19.180 INFO:teuthology.orchestra.run.smithi007.stdout:0 upgraded, 8 newly installed, 0 to remove and 98 not upgraded. 2022-03-20T14:50:19.180 INFO:teuthology.orchestra.run.smithi007.stdout:Need to get 77.7 MB of archives. 2022-03-20T14:50:19.180 INFO:teuthology.orchestra.run.smithi007.stdout:After this operation, 335 MB of additional disk space will be used. 2022-03-20T14:50:19.181 INFO:teuthology.orchestra.run.smithi007.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-20T14:50:19.352 INFO:teuthology.orchestra.run.smithi007.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-20T14:50:19.423 INFO:teuthology.orchestra.run.smithi007.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.23.0-00 [15.3 MB] 2022-03-20T14:50:19.461 INFO:teuthology.orchestra.run.smithi007.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-20T14:50:19.620 INFO:teuthology.orchestra.run.smithi087.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-20T14:50:19.672 INFO:teuthology.orchestra.run.smithi057.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-20T14:50:20.202 INFO:teuthology.orchestra.run.smithi007.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-20T14:50:20.358 INFO:teuthology.orchestra.run.smithi087.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.5-00 [19.5 MB] 2022-03-20T14:50:20.403 INFO:teuthology.orchestra.run.smithi057.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.5-00 [19.5 MB] 2022-03-20T14:50:20.838 INFO:teuthology.orchestra.run.smithi007.stdout:Get:4 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.5-00 [19.5 MB] 2022-03-20T14:50:21.259 INFO:teuthology.orchestra.run.smithi007.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.5-00 [8,931 kB] 2022-03-20T14:50:21.286 INFO:teuthology.orchestra.run.smithi087.stdout:Get:7 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.5-00 [8,931 kB] 2022-03-20T14:50:21.325 INFO:teuthology.orchestra.run.smithi057.stdout:Get:7 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.5-00 [8,931 kB] 2022-03-20T14:50:21.576 INFO:teuthology.orchestra.run.smithi087.stdout:Get:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.5-00 [8,582 kB] 2022-03-20T14:50:21.622 INFO:teuthology.orchestra.run.smithi007.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.5-00 [8,582 kB] 2022-03-20T14:50:21.671 INFO:teuthology.orchestra.run.smithi057.stdout:Get:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.5-00 [8,582 kB] 2022-03-20T14:50:22.213 INFO:teuthology.orchestra.run.smithi057.stderr:debconf: unable to initialize frontend: Dialog 2022-03-20T14:50:22.214 INFO:teuthology.orchestra.run.smithi057.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-20T14:50:22.214 INFO:teuthology.orchestra.run.smithi057.stderr:debconf: falling back to frontend: Readline 2022-03-20T14:50:22.221 INFO:teuthology.orchestra.run.smithi057.stderr:debconf: unable to initialize frontend: Readline 2022-03-20T14:50:22.221 INFO:teuthology.orchestra.run.smithi057.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-20T14:50:22.221 INFO:teuthology.orchestra.run.smithi057.stderr:debconf: falling back to frontend: Teletype 2022-03-20T14:50:22.226 INFO:teuthology.orchestra.run.smithi087.stderr:debconf: unable to initialize frontend: Dialog 2022-03-20T14:50:22.226 INFO:teuthology.orchestra.run.smithi087.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-20T14:50:22.227 INFO:teuthology.orchestra.run.smithi087.stderr:debconf: falling back to frontend: Readline 2022-03-20T14:50:22.227 INFO:teuthology.orchestra.run.smithi007.stderr:debconf: unable to initialize frontend: Dialog 2022-03-20T14:50:22.228 INFO:teuthology.orchestra.run.smithi007.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-20T14:50:22.228 INFO:teuthology.orchestra.run.smithi007.stderr:debconf: falling back to frontend: Readline 2022-03-20T14:50:22.228 INFO:teuthology.orchestra.run.smithi057.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-20T14:50:22.236 INFO:teuthology.orchestra.run.smithi087.stderr:debconf: unable to initialize frontend: Readline 2022-03-20T14:50:22.236 INFO:teuthology.orchestra.run.smithi087.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-20T14:50:22.236 INFO:teuthology.orchestra.run.smithi087.stderr:debconf: falling back to frontend: Teletype 2022-03-20T14:50:22.237 INFO:teuthology.orchestra.run.smithi007.stderr:debconf: unable to initialize frontend: Readline 2022-03-20T14:50:22.237 INFO:teuthology.orchestra.run.smithi007.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-20T14:50:22.238 INFO:teuthology.orchestra.run.smithi007.stderr:debconf: falling back to frontend: Teletype 2022-03-20T14:50:22.241 INFO:teuthology.orchestra.run.smithi087.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-20T14:50:22.242 INFO:teuthology.orchestra.run.smithi007.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-20T14:50:22.276 INFO:teuthology.orchestra.run.smithi057.stdout:Fetched 77.7 MB in 3s (23.5 MB/s) 2022-03-20T14:50:22.291 INFO:teuthology.orchestra.run.smithi007.stdout:Fetched 77.7 MB in 3s (25.9 MB/s) 2022-03-20T14:50:22.292 INFO:teuthology.orchestra.run.smithi087.stdout:Fetched 77.7 MB in 3s (22.3 MB/s) 2022-03-20T14:50:22.347 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package conntrack. 2022-03-20T14:50:22.361 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package conntrack. 2022-03-20T14:50:22.383 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package conntrack. 2022-03-20T14:50:22.392 INFO:teuthology.orchestra.run.smithi057.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-20T14:50:22.395 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-20T14:50:22.407 INFO:teuthology.orchestra.run.smithi007.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-20T14:50:22.410 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-20T14:50:22.427 INFO:teuthology.orchestra.run.smithi087.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-20T14:50:22.431 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-20T14:50:22.437 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-20T14:50:22.452 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-20T14:50:22.473 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-20T14:50:22.748 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package cri-tools. 2022-03-20T14:50:22.762 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../1-cri-tools_1.23.0-00_amd64.deb ... 2022-03-20T14:50:22.763 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package cri-tools. 2022-03-20T14:50:22.778 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../1-cri-tools_1.23.0-00_amd64.deb ... 2022-03-20T14:50:22.798 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking cri-tools (1.23.0-00) ... 2022-03-20T14:50:22.809 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package cri-tools. 2022-03-20T14:50:22.821 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking cri-tools (1.23.0-00) ... 2022-03-20T14:50:22.823 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../1-cri-tools_1.23.0-00_amd64.deb ... 2022-03-20T14:50:22.867 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking cri-tools (1.23.0-00) ... 2022-03-20T14:50:24.163 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package ebtables. 2022-03-20T14:50:24.176 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-20T14:50:24.191 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ebtables. 2022-03-20T14:50:24.195 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package ebtables. 2022-03-20T14:50:24.203 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-20T14:50:24.204 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-20T14:50:24.229 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-20T14:50:24.258 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-20T14:50:24.261 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-20T14:50:24.449 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-20T14:50:24.462 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-20T14:50:24.507 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-20T14:50:24.527 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-20T14:50:24.539 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-20T14:50:24.541 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-20T14:50:24.553 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-20T14:50:24.585 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-20T14:50:24.597 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-20T14:50:25.977 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package socat. 2022-03-20T14:50:25.991 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-20T14:50:26.012 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package socat. 2022-03-20T14:50:26.025 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-20T14:50:26.059 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package socat. 2022-03-20T14:50:26.062 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-20T14:50:26.071 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-20T14:50:26.108 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-20T14:50:26.151 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-20T14:50:26.441 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package kubelet. 2022-03-20T14:50:26.456 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../5-kubelet_1.23.5-00_amd64.deb ... 2022-03-20T14:50:26.499 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking kubelet (1.23.5-00) ... 2022-03-20T14:50:26.513 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package kubelet. 2022-03-20T14:50:26.527 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../5-kubelet_1.23.5-00_amd64.deb ... 2022-03-20T14:50:26.563 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package kubelet. 2022-03-20T14:50:26.570 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking kubelet (1.23.5-00) ... 2022-03-20T14:50:26.576 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../5-kubelet_1.23.5-00_amd64.deb ... 2022-03-20T14:50:26.613 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking kubelet (1.23.5-00) ... 2022-03-20T14:50:28.877 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package kubectl. 2022-03-20T14:50:28.889 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../6-kubectl_1.23.5-00_amd64.deb ... 2022-03-20T14:50:28.926 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking kubectl (1.23.5-00) ... 2022-03-20T14:50:28.989 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package kubectl. 2022-03-20T14:50:29.002 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../6-kubectl_1.23.5-00_amd64.deb ... 2022-03-20T14:50:29.031 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking kubectl (1.23.5-00) ... 2022-03-20T14:50:29.040 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package kubectl. 2022-03-20T14:50:29.053 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../6-kubectl_1.23.5-00_amd64.deb ... 2022-03-20T14:50:29.090 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking kubectl (1.23.5-00) ... 2022-03-20T14:50:30.080 INFO:teuthology.orchestra.run.smithi087.stdout:Selecting previously unselected package kubeadm. 2022-03-20T14:50:30.092 INFO:teuthology.orchestra.run.smithi087.stdout:Preparing to unpack .../7-kubeadm_1.23.5-00_amd64.deb ... 2022-03-20T14:50:30.121 INFO:teuthology.orchestra.run.smithi087.stdout:Unpacking kubeadm (1.23.5-00) ... 2022-03-20T14:50:30.236 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package kubeadm. 2022-03-20T14:50:30.251 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../7-kubeadm_1.23.5-00_amd64.deb ... 2022-03-20T14:50:30.270 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package kubeadm. 2022-03-20T14:50:30.283 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../7-kubeadm_1.23.5-00_amd64.deb ... 2022-03-20T14:50:30.311 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking kubeadm (1.23.5-00) ... 2022-03-20T14:50:30.361 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking kubeadm (1.23.5-00) ... 2022-03-20T14:50:31.236 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-20T14:50:31.345 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up kubectl (1.23.5-00) ... 2022-03-20T14:50:31.453 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-20T14:50:31.508 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-20T14:50:31.510 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-20T14:50:31.624 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up kubectl (1.23.5-00) ... 2022-03-20T14:50:31.632 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up kubectl (1.23.5-00) ... 2022-03-20T14:50:31.725 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-20T14:50:31.759 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-20T14:50:32.240 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-20T14:50:32.365 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up cri-tools (1.23.0-00) ... 2022-03-20T14:50:32.491 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-20T14:50:32.511 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-20T14:50:32.574 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-20T14:50:32.639 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up cri-tools (1.23.0-00) ... 2022-03-20T14:50:32.644 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up kubelet (1.23.5-00) ... 2022-03-20T14:50:32.680 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up cri-tools (1.23.0-00) ... 2022-03-20T14:50:32.757 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-20T14:50:32.808 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-20T14:50:32.868 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-20T14:50:32.893 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up kubelet (1.23.5-00) ... 2022-03-20T14:50:32.935 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up kubelet (1.23.5-00) ... 2022-03-20T14:50:33.112 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-20T14:50:33.147 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-20T14:50:33.372 INFO:teuthology.orchestra.run.smithi087.stdout:Setting up kubeadm (1.23.5-00) ... 2022-03-20T14:50:33.622 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up kubeadm (1.23.5-00) ... 2022-03-20T14:50:33.647 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up kubeadm (1.23.5-00) ... 2022-03-20T14:50:33.751 INFO:teuthology.orchestra.run.smithi087.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-20T14:50:34.001 INFO:teuthology.orchestra.run.smithi057.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-20T14:50:34.104 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-20T14:50:34.980 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-20T14:50:35.023 DEBUG:teuthology.orchestra.run.smithi057:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-20T14:50:35.027 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-20T14:50:48.471 INFO:teuthology.orchestra.run.smithi087.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.5 2022-03-20T14:50:48.610 INFO:teuthology.orchestra.run.smithi007.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.5 2022-03-20T14:50:49.319 INFO:teuthology.orchestra.run.smithi057.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.5 2022-03-20T14:50:51.698 INFO:teuthology.orchestra.run.smithi087.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.5 2022-03-20T14:50:52.244 INFO:teuthology.orchestra.run.smithi007.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.5 2022-03-20T14:50:52.280 INFO:teuthology.orchestra.run.smithi057.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.5 2022-03-20T14:50:53.921 INFO:teuthology.orchestra.run.smithi087.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.5 2022-03-20T14:50:54.458 INFO:teuthology.orchestra.run.smithi007.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.5 2022-03-20T14:50:54.753 INFO:teuthology.orchestra.run.smithi057.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.5 2022-03-20T14:50:57.312 INFO:teuthology.orchestra.run.smithi087.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.5 2022-03-20T14:50:58.078 INFO:teuthology.orchestra.run.smithi007.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.5 2022-03-20T14:50:58.377 INFO:teuthology.orchestra.run.smithi057.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.5 2022-03-20T14:50:58.628 INFO:teuthology.orchestra.run.smithi087.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-20T14:50:59.483 INFO:teuthology.orchestra.run.smithi007.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-20T14:50:59.701 INFO:teuthology.orchestra.run.smithi057.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-20T14:51:03.544 INFO:teuthology.orchestra.run.smithi087.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-20T14:51:05.091 INFO:teuthology.orchestra.run.smithi007.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-20T14:51:05.561 INFO:teuthology.orchestra.run.smithi087.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-20T14:51:05.749 INFO:teuthology.orchestra.run.smithi057.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-20T14:51:06.999 INFO:teuthology.orchestra.run.smithi007.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-20T14:51:07.733 INFO:teuthology.orchestra.run.smithi057.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-20T14:51:07.738 INFO:tasks.kubeadm:Token: abcdef.jtilwyxc1o88mtg6 2022-03-20T14:51:07.738 INFO:tasks.kubeadm:Remotes: {Remote(name='ubuntu@smithi007.front.sepia.ceph.com'): '172.21.15.7', Remote(name='ubuntu@smithi057.front.sepia.ceph.com'): '172.21.15.57', Remote(name='ubuntu@smithi087.front.sepia.ceph.com'): '172.21.15.87'} 2022-03-20T14:51:07.739 DEBUG:teuthology.orchestra.run.smithi007:> sudo kubeadm init --node-name smithi007 --token abcdef.jtilwyxc1o88mtg6 --pod-network-cidr 10.248.48.0/21 2022-03-20T14:51:07.951 INFO:teuthology.orchestra.run.smithi007.stdout:[init] Using Kubernetes version: v1.23.5 2022-03-20T14:51:07.951 INFO:teuthology.orchestra.run.smithi007.stdout:[preflight] Running pre-flight checks 2022-03-20T14:51:08.357 INFO:teuthology.orchestra.run.smithi007.stdout:[preflight] Pulling images required for setting up a Kubernetes cluster 2022-03-20T14:51:08.357 INFO:teuthology.orchestra.run.smithi007.stdout:[preflight] This might take a minute or two, depending on the speed of your internet connection 2022-03-20T14:51:08.357 INFO:teuthology.orchestra.run.smithi007.stdout:[preflight] You can also perform this action in beforehand using 'kubeadm config images pull' 2022-03-20T14:51:08.675 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] Using certificateDir folder "/etc/kubernetes/pki" 2022-03-20T14:51:08.896 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] Generating "ca" certificate and key 2022-03-20T14:51:09.150 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] Generating "apiserver" certificate and key 2022-03-20T14:51:09.150 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local smithi007] and IPs [10.96.0.1 172.21.15.7] 2022-03-20T14:51:09.571 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] Generating "apiserver-kubelet-client" certificate and key 2022-03-20T14:51:10.023 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] Generating "front-proxy-ca" certificate and key 2022-03-20T14:51:10.423 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] Generating "front-proxy-client" certificate and key 2022-03-20T14:51:10.620 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] Generating "etcd/ca" certificate and key 2022-03-20T14:51:10.697 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] Generating "etcd/server" certificate and key 2022-03-20T14:51:10.698 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] etcd/server serving cert is signed for DNS names [localhost smithi007] and IPs [172.21.15.7 127.0.0.1 ::1] 2022-03-20T14:51:10.852 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] Generating "etcd/peer" certificate and key 2022-03-20T14:51:10.853 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] etcd/peer serving cert is signed for DNS names [localhost smithi007] and IPs [172.21.15.7 127.0.0.1 ::1] 2022-03-20T14:51:10.990 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] Generating "etcd/healthcheck-client" certificate and key 2022-03-20T14:51:11.093 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] Generating "apiserver-etcd-client" certificate and key 2022-03-20T14:51:11.260 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] Generating "sa" key and public key 2022-03-20T14:51:11.261 INFO:teuthology.orchestra.run.smithi007.stdout:[kubeconfig] Using kubeconfig folder "/etc/kubernetes" 2022-03-20T14:51:11.437 INFO:teuthology.orchestra.run.smithi007.stdout:[kubeconfig] Writing "admin.conf" kubeconfig file 2022-03-20T14:51:11.602 INFO:teuthology.orchestra.run.smithi007.stdout:[kubeconfig] Writing "kubelet.conf" kubeconfig file 2022-03-20T14:51:11.791 INFO:teuthology.orchestra.run.smithi007.stdout:[kubeconfig] Writing "controller-manager.conf" kubeconfig file 2022-03-20T14:51:12.178 INFO:teuthology.orchestra.run.smithi007.stdout:[kubeconfig] Writing "scheduler.conf" kubeconfig file 2022-03-20T14:51:12.194 INFO:teuthology.orchestra.run.smithi007.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-20T14:51:12.195 INFO:teuthology.orchestra.run.smithi007.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-20T14:51:12.195 INFO:teuthology.orchestra.run.smithi007.stdout:[kubelet-start] Starting the kubelet 2022-03-20T14:51:12.434 INFO:teuthology.orchestra.run.smithi007.stdout:[control-plane] Using manifest folder "/etc/kubernetes/manifests" 2022-03-20T14:51:12.435 INFO:teuthology.orchestra.run.smithi007.stdout:[control-plane] Creating static Pod manifest for "kube-apiserver" 2022-03-20T14:51:12.438 INFO:teuthology.orchestra.run.smithi007.stdout:[control-plane] Creating static Pod manifest for "kube-controller-manager" 2022-03-20T14:51:12.438 INFO:teuthology.orchestra.run.smithi007.stdout:[control-plane] Creating static Pod manifest for "kube-scheduler" 2022-03-20T14:51:12.439 INFO:teuthology.orchestra.run.smithi007.stdout:[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" 2022-03-20T14:51:12.440 INFO:teuthology.orchestra.run.smithi007.stdout:[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s 2022-03-20T14:51:29.944 INFO:teuthology.orchestra.run.smithi007.stdout:[apiclient] All control plane components are healthy after 17.503710 seconds 2022-03-20T14:51:29.945 INFO:teuthology.orchestra.run.smithi007.stdout:[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace 2022-03-20T14:51:29.966 INFO:teuthology.orchestra.run.smithi007.stdout:[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster 2022-03-20T14:51:29.967 INFO:teuthology.orchestra.run.smithi007.stdout:NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently. 2022-03-20T14:51:30.523 INFO:teuthology.orchestra.run.smithi007.stdout:[upload-certs] Skipping phase. Please see --upload-certs 2022-03-20T14:51:30.523 INFO:teuthology.orchestra.run.smithi007.stdout:[mark-control-plane] Marking the node smithi007 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] 2022-03-20T14:51:30.523 INFO:teuthology.orchestra.run.smithi007.stdout:[mark-control-plane] Marking the node smithi007 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] 2022-03-20T14:51:31.062 INFO:teuthology.orchestra.run.smithi007.stdout:[bootstrap-token] Using token: abcdef.jtilwyxc1o88mtg6 2022-03-20T14:51:31.063 INFO:teuthology.orchestra.run.smithi007.stdout:[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles 2022-03-20T14:51:31.109 INFO:teuthology.orchestra.run.smithi007.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes 2022-03-20T14:51:31.124 INFO:teuthology.orchestra.run.smithi007.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials 2022-03-20T14:51:31.151 INFO:teuthology.orchestra.run.smithi007.stdout:[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token 2022-03-20T14:51:31.214 INFO:teuthology.orchestra.run.smithi007.stdout:[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster 2022-03-20T14:51:31.217 INFO:teuthology.orchestra.run.smithi007.stdout:[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace 2022-03-20T14:51:31.257 INFO:teuthology.orchestra.run.smithi007.stdout:[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key 2022-03-20T14:51:31.708 INFO:teuthology.orchestra.run.smithi007.stdout:[addons] Applied essential addon: CoreDNS 2022-03-20T14:51:31.887 INFO:teuthology.orchestra.run.smithi007.stdout:[addons] Applied essential addon: kube-proxy 2022-03-20T14:51:31.889 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T14:51:31.889 INFO:teuthology.orchestra.run.smithi007.stdout:Your Kubernetes control-plane has initialized successfully! 2022-03-20T14:51:31.889 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T14:51:31.889 INFO:teuthology.orchestra.run.smithi007.stdout:To start using your cluster, you need to run the following as a regular user: 2022-03-20T14:51:31.890 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T14:51:31.890 INFO:teuthology.orchestra.run.smithi007.stdout: mkdir -p $HOME/.kube 2022-03-20T14:51:31.890 INFO:teuthology.orchestra.run.smithi007.stdout: sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 2022-03-20T14:51:31.890 INFO:teuthology.orchestra.run.smithi007.stdout: sudo chown $(id -u):$(id -g) $HOME/.kube/config 2022-03-20T14:51:31.891 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T14:51:31.891 INFO:teuthology.orchestra.run.smithi007.stdout:Alternatively, if you are the root user, you can run: 2022-03-20T14:51:31.891 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T14:51:31.891 INFO:teuthology.orchestra.run.smithi007.stdout: export KUBECONFIG=/etc/kubernetes/admin.conf 2022-03-20T14:51:31.892 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T14:51:31.892 INFO:teuthology.orchestra.run.smithi007.stdout:You should now deploy a pod network to the cluster. 2022-03-20T14:51:31.892 INFO:teuthology.orchestra.run.smithi007.stdout:Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: 2022-03-20T14:51:31.893 INFO:teuthology.orchestra.run.smithi007.stdout: https://kubernetes.io/docs/concepts/cluster-administration/addons/ 2022-03-20T14:51:31.893 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T14:51:31.893 INFO:teuthology.orchestra.run.smithi007.stdout:Then you can join any number of worker nodes by running the following on each as root: 2022-03-20T14:51:31.894 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T14:51:31.894 INFO:teuthology.orchestra.run.smithi007.stdout:kubeadm join 172.21.15.7:6443 --token abcdef.jtilwyxc1o88mtg6 \ 2022-03-20T14:51:31.894 INFO:teuthology.orchestra.run.smithi007.stdout: --discovery-token-ca-cert-hash sha256:1da97860363d5eddd24ec33f1ef741f82b3c286b75f036da3a06300c5f2cfda8 2022-03-20T14:51:31.896 DEBUG:teuthology.orchestra.run.smithi057:> sudo kubeadm join 172.21.15.7:6443 --node-name smithi057 --token abcdef.jtilwyxc1o88mtg6 --discovery-token-unsafe-skip-ca-verification 2022-03-20T14:51:31.899 DEBUG:teuthology.orchestra.run.smithi087:> sudo kubeadm join 172.21.15.7:6443 --node-name smithi087 --token abcdef.jtilwyxc1o88mtg6 --discovery-token-unsafe-skip-ca-verification 2022-03-20T14:51:31.937 INFO:teuthology.orchestra.run.smithi057.stdout:[preflight] Running pre-flight checks 2022-03-20T14:51:31.945 INFO:teuthology.orchestra.run.smithi087.stdout:[preflight] Running pre-flight checks 2022-03-20T14:51:44.783 INFO:teuthology.orchestra.run.smithi057.stdout:[preflight] Reading configuration from the cluster... 2022-03-20T14:51:44.783 INFO:teuthology.orchestra.run.smithi057.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-03-20T14:51:45.220 INFO:teuthology.orchestra.run.smithi057.stderr:W0320 14:51:45.220273 11124 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf 2022-03-20T14:51:45.238 INFO:teuthology.orchestra.run.smithi057.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-20T14:51:45.238 INFO:teuthology.orchestra.run.smithi057.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-20T14:51:45.238 INFO:teuthology.orchestra.run.smithi057.stdout:[kubelet-start] Starting the kubelet 2022-03-20T14:51:45.449 INFO:teuthology.orchestra.run.smithi057.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-03-20T14:51:45.666 INFO:teuthology.orchestra.run.smithi087.stdout:[preflight] Reading configuration from the cluster... 2022-03-20T14:51:45.666 INFO:teuthology.orchestra.run.smithi087.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-03-20T14:51:46.029 INFO:teuthology.orchestra.run.smithi087.stderr:W0320 14:51:46.030146 11122 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf 2022-03-20T14:51:46.045 INFO:teuthology.orchestra.run.smithi087.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-20T14:51:46.045 INFO:teuthology.orchestra.run.smithi087.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-20T14:51:46.046 INFO:teuthology.orchestra.run.smithi087.stdout:[kubelet-start] Starting the kubelet 2022-03-20T14:51:46.274 INFO:teuthology.orchestra.run.smithi087.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-03-20T14:51:58.971 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-20T14:51:58.971 INFO:teuthology.orchestra.run.smithi057.stdout:This node has joined the cluster: 2022-03-20T14:51:58.971 INFO:teuthology.orchestra.run.smithi057.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-03-20T14:51:58.971 INFO:teuthology.orchestra.run.smithi057.stdout:* The Kubelet was informed of the new secure connection details. 2022-03-20T14:51:58.972 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-20T14:51:58.972 INFO:teuthology.orchestra.run.smithi057.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-03-20T14:51:58.972 INFO:teuthology.orchestra.run.smithi057.stdout: 2022-03-20T14:52:00.295 INFO:teuthology.orchestra.run.smithi087.stdout: 2022-03-20T14:52:00.295 INFO:teuthology.orchestra.run.smithi087.stdout:This node has joined the cluster: 2022-03-20T14:52:00.296 INFO:teuthology.orchestra.run.smithi087.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-03-20T14:52:00.296 INFO:teuthology.orchestra.run.smithi087.stdout:* The Kubelet was informed of the new secure connection details. 2022-03-20T14:52:00.296 INFO:teuthology.orchestra.run.smithi087.stdout: 2022-03-20T14:52:00.296 INFO:teuthology.orchestra.run.smithi087.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-03-20T14:52:00.297 INFO:teuthology.orchestra.run.smithi087.stdout: 2022-03-20T14:52:00.302 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-20T14:52:00.302 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/etc/kubernetes/admin.conf of=/dev/stdout 2022-03-20T14:52:00.317 INFO:tasks.kubeadm:Setting up kubectl 2022-03-20T14:52:00.318 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-20T14:52:00.333 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-20T14:52:00.351 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-20T14:52:00.367 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-20T14:52:00.368 DEBUG:teuthology.orchestra.run.smithi007:> dd of=.kube/config 2022-03-20T14:52:00.375 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-20T14:52:00.376 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/root/.kube/config 2022-03-20T14:52:00.435 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2022-03-20T14:52:00.436 DEBUG:teuthology.orchestra.run.smithi057:> dd of=.kube/config 2022-03-20T14:52:00.444 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2022-03-20T14:52:00.444 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/root/.kube/config 2022-03-20T14:52:00.503 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2022-03-20T14:52:00.504 DEBUG:teuthology.orchestra.run.smithi087:> dd of=.kube/config 2022-03-20T14:52:00.511 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2022-03-20T14:52:00.511 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/root/.kube/config 2022-03-20T14:52:00.571 DEBUG:teuthology.orchestra.run.smithi007:> curl https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml 2022-03-20T14:52:00.587 INFO:teuthology.orchestra.run.smithi007.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-03-20T14:52:00.587 INFO:teuthology.orchestra.run.smithi007.stderr: Dload Upload Total Spent Left Speed 2022-03-20T14:52:00.726 INFO:teuthology.orchestra.run.smithi007.stdout:--- 2022-03-20T14:52:00.727 INFO:teuthology.orchestra.run.smithi007.stdout:apiVersion: policy/v1beta1 2022-03-20T14:52:00.727 INFO:teuthology.orchestra.run.smithi007.stdout:kind: PodSecurityPolicy 2022-03-20T14:52:00.727 INFO:teuthology.orchestra.run.smithi007.stdout:metadata: 2022-03-20T14:52:00.727 INFO:teuthology.orchestra.run.smithi007.stdout: name: psp.flannel.unprivileged 2022-03-20T14:52:00.728 INFO:teuthology.orchestra.run.smithi007.stdout: annotations: 2022-03-20T14:52:00.728 INFO:teuthology.orchestra.run.smithi007.stdout: seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default 2022-03-20T14:52:00.728 INFO:teuthology.orchestra.run.smithi007.stdout: seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default 2022-03-20T14:52:00.728 INFO:teuthology.orchestra.run.smithi007.stdout: apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default 2022-03-20T14:52:00.729 INFO:teuthology.orchestra.run.smithi007.stdout: apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default 2022-03-20T14:52:00.729 INFO:teuthology.orchestra.run.smithi007.stdout:spec: 2022-03-20T14:52:00.729 INFO:teuthology.orchestra.run.smithi007.stdout: privileged: false 2022-03-20T14:52:00.729 INFO:teuthology.orchestra.run.smithi007.stdout: volumes: 2022-03-20T14:52:00.729 INFO:teuthology.orchestra.run.smithi007.stdout: - configMap 2022-03-20T14:52:00.730 INFO:teuthology.orchestra.run.smithi007.stdout: - secret 2022-03-20T14:52:00.730 INFO:teuthology.orchestra.run.smithi007.stdout: - emptyDir 2022-03-20T14:52:00.730 INFO:teuthology.orchestra.run.smithi007.stdout: - hostPath 2022-03-20T14:52:00.730 INFO:teuthology.orchestra.run.smithi007.stdout: allowedHostPaths: 2022-03-20T14:52:00.731 INFO:teuthology.orchestra.run.smithi007.stdout: - pathPrefix: "/etc/cni/net.d" 2022-03-20T14:52:00.731 INFO:teuthology.orchestra.run.smithi007.stdout: - pathPrefix: "/etc/kube-flannel" 2022-03-20T14:52:00.731 INFO:teuthology.orchestra.run.smithi007.stdout: - pathPrefix: "/run/flannel" 2022-03-20T14:52:00.731 INFO:teuthology.orchestra.run.smithi007.stdout: readOnlyRootFilesystem: false 2022-03-20T14:52:00.731 INFO:teuthology.orchestra.run.smithi007.stdout: # Users and groups 2022-03-20T14:52:00.732 INFO:teuthology.orchestra.run.smithi007.stdout: runAsUser: 2022-03-20T14:52:00.732 INFO:teuthology.orchestra.run.smithi007.stdout: rule: RunAsAny 2022-03-20T14:52:00.732 INFO:teuthology.orchestra.run.smithi007.stdout: supplementalGroups: 2022-03-20T14:52:00.733 INFO:teuthology.orchestra.run.smithi007.stdout: rule: RunAsAny 2022-03-20T14:52:00.733 INFO:teuthology.orchestra.run.smithi007.stdout: fsGroup: 2022-03-20T14:52:00.733 INFO:teuthology.orchestra.run.smithi007.stdout: rule: RunAsAny 2022-03-20T14:52:00.734 INFO:teuthology.orchestra.run.smithi007.stdout: # Privilege Escalation 2022-03-20T14:52:00.734 INFO:teuthology.orchestra.run.smithi007.stdout: allowPrivilegeEscalation: false 2022-03-20T14:52:00.734 INFO:teuthology.orchestra.run.smithi007.stdout: defaultAllowPrivilegeEscalation: false 2022-03-20T14:52:00.734 INFO:teuthology.orchestra.run.smithi007.stdout: # Capabilities 2022-03-20T14:52:00.734 INFO:teuthology.orchestra.run.smithi007.stdout: allowedCapabilities: ['NET_ADMIN', 'NET_RAW'] 2022-03-20T14:52:00.735 INFO:teuthology.orchestra.run.smithi007.stdout: defaultAddCapabilities: [] 2022-03-20T14:52:00.735 INFO:teuthology.orchestra.run.smithi007.stdout: requiredDropCapabilities: [] 2022-03-20T14:52:00.735 INFO:teuthology.orchestra.run.smithi007.stdout: # Host namespaces 2022-03-20T14:52:00.735 INFO:teuthology.orchestra.run.smithi007.stdout: hostPID: false 2022-03-20T14:52:00.736 INFO:teuthology.orchestra.run.smithi007.stdout: hostIPC: false 2022-03-20T14:52:00.736 INFO:teuthology.orchestra.run.smithi007.stdout: hostNetwork: true 2022-03-20T14:52:00.736 INFO:teuthology.orchestra.run.smithi007.stdout: hostPorts: 2022-03-20T14:52:00.736 INFO:teuthology.orchestra.run.smithi007.stdout: - min: 0 2022-03-20T14:52:00.736 INFO:teuthology.orchestra.run.smithi007.stdout: max: 65535 2022-03-20T14:52:00.737 INFO:teuthology.orchestra.run.smithi007.stdout: # SELinux 2022-03-20T14:52:00.737 INFO:teuthology.orchestra.run.smithi007.stdout: seLinux: 2022-03-20T14:52:00.737 INFO:teuthology.orchestra.run.smithi007.stdout: # SELinux is unused in CaaSP 2022-03-20T14:52:00.737 INFO:teuthology.orchestra.run.smithi007.stdout: rule: 'RunAsAny' 2022-03-20T14:52:00.738 INFO:teuthology.orchestra.run.smithi007.stdout:--- 2022-03-20T14:52:00.738 INFO:teuthology.orchestra.run.smithi007.stdout:kind: ClusterRole 2022-03-20T14:52:00.738 INFO:teuthology.orchestra.run.smithi007.stdout:apiVersion: rbac.authorization.k8s.io/v1 2022-03-20T14:52:00.738 INFO:teuthology.orchestra.run.smithi007.stdout:metadata: 2022-03-20T14:52:00.738 INFO:teuthology.orchestra.run.smithi007.stdout: name: flannel 2022-03-20T14:52:00.739 INFO:teuthology.orchestra.run.smithi007.stdout:rules: 2022-03-20T14:52:00.739 INFO:teuthology.orchestra.run.smithi007.stdout:- apiGroups: ['extensions'] 2022-03-20T14:52:00.739 INFO:teuthology.orchestra.run.smithi007.stdout: resources: ['podsecuritypolicies'] 2022-03-20T14:52:00.739 INFO:teuthology.orchestra.run.smithi007.stdout: verbs: ['use'] 2022-03-20T14:52:00.740 INFO:teuthology.orchestra.run.smithi007.stdout: resourceNames: ['psp.flannel.unprivileged'] 2022-03-20T14:52:00.740 INFO:teuthology.orchestra.run.smithi007.stdout:- apiGroups: 2022-03-20T14:52:00.740 INFO:teuthology.orchestra.run.smithi007.stdout: - "" 2022-03-20T14:52:00.740 INFO:teuthology.orchestra.run.smithi007.stdout: resources: 2022-03-20T14:52:00.740 INFO:teuthology.orchestra.run.smithi007.stdout: - pods 2022-03-20T14:52:00.741 INFO:teuthology.orchestra.run.smithi007.stdout: verbs: 2022-03-20T14:52:00.741 INFO:teuthology.orchestra.run.smithi007.stdout: - get 2022-03-20T14:52:00.741 INFO:teuthology.orchestra.run.smithi007.stdout:- apiGroups: 2022-03-20T14:52:00.741 INFO:teuthology.orchestra.run.smithi007.stdout: - "" 2022-03-20T14:52:00.742 INFO:teuthology.orchestra.run.smithi007.stdout: resources: 2022-03-20T14:52:00.742 INFO:teuthology.orchestra.run.smithi007.stdout: - nodes 2022-03-20T14:52:00.742 INFO:teuthology.orchestra.run.smithi007.stdout: verbs: 2022-03-20T14:52:00.742 INFO:teuthology.orchestra.run.smithi007.stdout: - list 2022-03-20T14:52:00.742 INFO:teuthology.orchestra.run.smithi007.stdout: - watch 2022-03-20T14:52:00.743 INFO:teuthology.orchestra.run.smithi007.stdout:- apiGroups: 2022-03-20T14:52:00.743 INFO:teuthology.orchestra.run.smithi007.stdout: - "" 2022-03-20T14:52:00.743 INFO:teuthology.orchestra.run.smithi007.stdout: resources: 2022-03-20T14:52:00.743 INFO:teuthology.orchestra.run.smithi007.stdout: - nodes/status 2022-03-20T14:52:00.744 INFO:teuthology.orchestra.run.smithi007.stdout: verbs: 2022-03-20T14:52:00.744 INFO:teuthology.orchestra.run.smithi007.stdout: - patch 2022-03-20T14:52:00.744 INFO:teuthology.orchestra.run.smithi007.stdout:--- 2022-03-20T14:52:00.744 INFO:teuthology.orchestra.run.smithi007.stdout:kind: ClusterRoleBinding 2022-03-20T14:52:00.745 INFO:teuthology.orchestra.run.smithi007.stdout:apiVersion: rbac.authorization.k8s.io/v1 2022-03-20T14:52:00.745 INFO:teuthology.orchestra.run.smithi007.stdout:metadata: 2022-03-20T14:52:00.745 INFO:teuthology.orchestra.run.smithi007.stdout: name: flannel 2022-03-20T14:52:00.745 INFO:teuthology.orchestra.run.smithi007.stdout:roleRef: 2022-03-20T14:52:00.745 INFO:teuthology.orchestra.run.smithi007.stdout: apiGroup: rbac.authorization.k8s.io 2022-03-20T14:52:00.746 INFO:teuthology.orchestra.run.smithi007.stdout: kind: ClusterRole 2022-03-20T14:52:00.746 INFO:teuthology.orchestra.run.smithi007.stdout: name: flannel 2022-03-20T14:52:00.746 INFO:teuthology.orchestra.run.smithi007.stdout:subjects: 2022-03-20T14:52:00.746 INFO:teuthology.orchestra.run.smithi007.stdout:- kind: ServiceAccount 2022-03-20T14:52:00.747 INFO:teuthology.orchestra.run.smithi007.stdout: name: flannel 2022-03-20T14:52:00.747 INFO:teuthology.orchestra.run.smithi007.stdout: namespace: kube-system 2022-03-20T14:52:00.747 INFO:teuthology.orchestra.run.smithi007.stdout:--- 2022-03-20T14:52:00.747 INFO:teuthology.orchestra.run.smithi007.stdout:apiVersion: v1 2022-03-20T14:52:00.747 INFO:teuthology.orchestra.run.smithi007.stdout:kind: ServiceAccount 2022-03-20T14:52:00.748 INFO:teuthology.orchestra.run.smithi007.stdout:metadata: 2022-03-20T14:52:00.748 INFO:teuthology.orchestra.run.smithi007.stdout: name: flannel 2022-03-20T14:52:00.748 INFO:teuthology.orchestra.run.smithi007.stdout: namespace: kube-system 2022-03-20T14:52:00.748 INFO:teuthology.orchestra.run.smithi007.stdout:--- 2022-03-20T14:52:00.749 INFO:teuthology.orchestra.run.smithi007.stdout:kind: ConfigMap 2022-03-20T14:52:00.749 INFO:teuthology.orchestra.run.smithi007.stdout:apiVersion: v1 2022-03-20T14:52:00.749 INFO:teuthology.orchestra.run.smithi007.stdout:metadata: 2022-03-20T14:52:00.749 INFO:teuthology.orchestra.run.smithi007.stdout: name: kube-flannel-cfg 2022-03-20T14:52:00.750 INFO:teuthology.orchestra.run.smithi007.stdout: namespace: kube-system 2022-03-20T14:52:00.750 INFO:teuthology.orchestra.run.smithi007.stdout: labels: 2022-03-20T14:52:00.750 INFO:teuthology.orchestra.run.smithi007.stdout: tier: node 2022-03-20T14:52:00.750 INFO:teuthology.orchestra.run.smithi007.stdout: app: flannel 2022-03-20T14:52:00.750 INFO:teuthology.orchestra.run.smithi007.stdout:data: 2022-03-20T14:52:00.751 INFO:teuthology.orchestra.run.smithi007.stdout: cni-conf.json: | 2022-03-20T14:52:00.751 INFO:teuthology.orchestra.run.smithi007.stdout: { 2022-03-20T14:52:00.751 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "cbr0", 2022-03-20T14:52:00.751 INFO:teuthology.orchestra.run.smithi007.stdout: "cniVersion": "0.3.1", 2022-03-20T14:52:00.752 INFO:teuthology.orchestra.run.smithi007.stdout: "plugins": [ 2022-03-20T14:52:00.752 INFO:teuthology.orchestra.run.smithi007.stdout: { 2022-03-20T14:52:00.752 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "flannel", 2022-03-20T14:52:00.752 INFO:teuthology.orchestra.run.smithi007.stdout: "delegate": { 2022-03-20T14:52:00.753 INFO:teuthology.orchestra.run.smithi007.stdout: "hairpinMode": true, 2022-03-20T14:52:00.753 INFO:teuthology.orchestra.run.smithi007.stdout: "isDefaultGateway": true 2022-03-20T14:52:00.753 INFO:teuthology.orchestra.run.smithi007.stdout: } 2022-03-20T14:52:00.753 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2022-03-20T14:52:00.753 INFO:teuthology.orchestra.run.smithi007.stdout: { 2022-03-20T14:52:00.754 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "portmap", 2022-03-20T14:52:00.754 INFO:teuthology.orchestra.run.smithi007.stdout: "capabilities": { 2022-03-20T14:52:00.754 INFO:teuthology.orchestra.run.smithi007.stdout: "portMappings": true 2022-03-20T14:52:00.754 INFO:teuthology.orchestra.run.smithi007.stdout: } 2022-03-20T14:52:00.755 INFO:teuthology.orchestra.run.smithi007.stdout: } 2022-03-20T14:52:00.755 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2022-03-20T14:52:00.755 INFO:teuthology.orchestra.run.smithi007.stdout: } 2022-03-20T14:52:00.756 INFO:teuthology.orchestra.run.smithi007.stdout: net-conf.json: | 2022-03-20T14:52:00.756 INFO:teuthology.orchestra.run.smithi007.stdout: { 2022-03-20T14:52:00.756 INFO:teuthology.orchestra.run.smithi007.stdout: "Network": "10.244.0.0/16", 2022-03-20T14:52:00.756 INFO:teuthology.orchestra.run.smithi007.stdout: "Backend": { 2022-03-20T14:52:00.757 INFO:teuthology.orchestra.run.smithi007.stdout: "Type": "vxlan" 2022-03-20T14:52:00.757 INFO:teuthology.orchestra.run.smithi007.stdout: } 2022-03-20T14:52:00.757 INFO:teuthology.orchestra.run.smithi007.stdout: } 2022-03-20T14:52:00.757 INFO:teuthology.orchestra.run.smithi007.stdout:--- 2022-03-20T14:52:00.757 INFO:teuthology.orchestra.run.smithi007.stdout:apiVersion: apps/v1 2022-03-20T14:52:00.758 INFO:teuthology.orchestra.run.smithi007.stdout:kind: DaemonSet 2022-03-20T14:52:00.758 INFO:teuthology.orchestra.run.smithi007.stdout:metadata: 2022-03-20T14:52:00.758 INFO:teuthology.orchestra.run.smithi007.stdout: name: kube-flannel-ds 2022-03-20T14:52:00.758 INFO:teuthology.orchestra.run.smithi007.stdout: namespace: kube-system 2022-03-20T14:52:00.759 INFO:teuthology.orchestra.run.smithi007.stdout: labels: 2022-03-20T14:52:00.759 INFO:teuthology.orchestra.run.smithi007.stdout: tier: node 2022-03-20T14:52:00.759 INFO:teuthology.orchestra.run.smithi007.stdout: app: flannel 2022-03-20T14:52:00.759 INFO:teuthology.orchestra.run.smithi007.stdout:spec: 2022-03-20T14:52:00.759 INFO:teuthology.orchestra.run.smithi007.stdout: selector: 2022-03-20T14:52:00.760 INFO:teuthology.orchestra.run.smithi007.stdout: matchLabels: 2022-03-20T14:52:00.760 INFO:teuthology.orchestra.run.smithi007.stdout: app: flannel 2022-03-20T14:52:00.760 INFO:teuthology.orchestra.run.smithi007.stdout: template: 2022-03-20T14:52:00.760 INFO:teuthology.orchestra.run.smithi007.stdout: metadata: 2022-03-20T14:52:00.761 INFO:teuthology.orchestra.run.smithi007.stdout: labels: 2022-03-20T14:52:00.761 INFO:teuthology.orchestra.run.smithi007.stdout: tier: node 2022-03-20T14:52:00.761 INFO:teuthology.orchestra.run.smithi007.stdout: app: flannel 2022-03-20T14:52:00.761 INFO:teuthology.orchestra.run.smithi007.stdout: spec: 2022-03-20T14:52:00.762 INFO:teuthology.orchestra.run.smithi007.stdout: affinity: 2022-03-20T14:52:00.762 INFO:teuthology.orchestra.run.smithi007.stdout: nodeAffinity: 2022-03-20T14:52:00.762 INFO:teuthology.orchestra.run.smithi007.stdout: requiredDuringSchedulingIgnoredDuringExecution: 2022-03-20T14:52:00.762 INFO:teuthology.orchestra.run.smithi007.stdout: nodeSelectorTerms: 2022-03-20T14:52:00.762 INFO:teuthology.orchestra.run.smithi007.stdout: - matchExpressions: 2022-03-20T14:52:00.763 INFO:teuthology.orchestra.run.smithi007.stdout: - key: kubernetes.io/os 2022-03-20T14:52:00.763 INFO:teuthology.orchestra.run.smithi007.stdout: operator: In 2022-03-20T14:52:00.763 INFO:teuthology.orchestra.run.smithi007.stdout: values: 2022-03-20T14:52:00.763 INFO:teuthology.orchestra.run.smithi007.stdout: - linux 2022-03-20T14:52:00.764 INFO:teuthology.orchestra.run.smithi007.stdout: hostNetwork: true 2022-03-20T14:52:00.764 INFO:teuthology.orchestra.run.smithi007.stdout: priorityClassName: system-node-critical 2022-03-20T14:52:00.764 INFO:teuthology.orchestra.run.smithi007.stdout: tolerations: 2022-03-20T14:52:00.764 INFO:teuthology.orchestra.run.smithi007.stdout: - operator: Exists 2022-03-20T14:52:00.764 INFO:teuthology.orchestra.run.smithi007.stdout: effect: NoSchedule 2022-03-20T14:52:00.765 INFO:teuthology.orchestra.run.smithi007.stdout: serviceAccountName: flannel 2022-03-20T14:52:00.765 INFO:teuthology.orchestra.run.smithi007.stdout: initContainers: 2022-03-20T14:52:00.765 INFO:teuthology.orchestra.run.smithi007.stdout: - name: install-cni-plugin 2022-03-20T14:52:00.765 INFO:teuthology.orchestra.run.smithi007.stdout: #image: flannelcni/flannel-cni-plugin:v1.0.1 for ppc64le and mips64le (dockerhub limitations may apply) 2022-03-20T14:52:00.766 INFO:teuthology.orchestra.run.smithi007.stdout: image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.1 2022-03-20T14:52:00.766 INFO:teuthology.orchestra.run.smithi007.stdout: command: 2022-03-20T14:52:00.766 INFO:teuthology.orchestra.run.smithi007.stdout: - cp 2022-03-20T14:52:00.766 INFO:teuthology.orchestra.run.smithi007.stdout: args: 2022-03-20T14:52:00.767 INFO:teuthology.orchestra.run.smithi007.stdout: - -f 2022-03-20T14:52:00.767 INFO:teuthology.orchestra.run.smithi007.stdout: - /flannel 2022-03-20T14:52:00.767 INFO:teuthology.orchestra.run.smithi007.stdout: - /opt/cni/bin/flannel 2022-03-20T14:52:00.767 INFO:teuthology.orchestra.run.smithi007.stdout: volumeMounts: 2022-03-20T14:52:00.767 INFO:teuthology.orchestra.run.smithi007.stdout: - name: cni-plugin 2022-03-20T14:52:00.768 INFO:teuthology.orchestra.run.smithi007.stdout: mountPath: /opt/cni/bin 2022-03-20T14:52:00.768 INFO:teuthology.orchestra.run.smithi007.stdout: - name: install-cni 2022-03-20T14:52:00.768 INFO:teuthology.orchestra.run.smithi007.stdout: #image: flannelcni/flannel:v0.17.0 for ppc64le and mips64le (dockerhub limitations may apply) 2022-03-20T14:52:00.768 INFO:teuthology.orchestra.run.smithi007.stdout: image: rancher/mirrored-flannelcni-flannel:v0.17.0 2022-03-20T14:52:00.769 INFO:teuthology.orchestra.run.smithi007.stdout: command: 2022-03-20T14:52:00.769 INFO:teuthology.orchestra.run.smithi007.stdout: - cp 2022-03-20T14:52:00.769 INFO:teuthology.orchestra.run.smithi007.stdout: args: 2022-03-20T14:52:00.769 INFO:teuthology.orchestra.run.smithi007.stdout: - -f 2022-03-20T14:52:00.769 INFO:teuthology.orchestra.run.smithi007.stdout: - /etc/kube-flannel/cni-conf.json 2022-03-20T14:52:00.771 INFO:teuthology.orchestra.run.smithi007.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 5692 100 5692 0 0 40657 0 --:--:-- --:--:-- --:--:-- 40368 100 5692 100 5692 0 0 40657 0 --:--:-- --:--:-- --:--:-- 40368 2022-03-20T14:52:00.772 INFO:teuthology.orchestra.run.smithi007.stdout: - /etc/cni/net.d/10-flannel.conflist 2022-03-20T14:52:00.772 INFO:teuthology.orchestra.run.smithi007.stdout: volumeMounts: 2022-03-20T14:52:00.772 INFO:teuthology.orchestra.run.smithi007.stdout: - name: cni 2022-03-20T14:52:00.772 INFO:teuthology.orchestra.run.smithi007.stdout: mountPath: /etc/cni/net.d 2022-03-20T14:52:00.773 INFO:teuthology.orchestra.run.smithi007.stdout: - name: flannel-cfg 2022-03-20T14:52:00.773 INFO:teuthology.orchestra.run.smithi007.stdout: mountPath: /etc/kube-flannel/ 2022-03-20T14:52:00.773 INFO:teuthology.orchestra.run.smithi007.stdout: containers: 2022-03-20T14:52:00.773 INFO:teuthology.orchestra.run.smithi007.stdout: - name: kube-flannel 2022-03-20T14:52:00.774 INFO:teuthology.orchestra.run.smithi007.stdout: #image: flannelcni/flannel:v0.17.0 for ppc64le and mips64le (dockerhub limitations may apply) 2022-03-20T14:52:00.774 INFO:teuthology.orchestra.run.smithi007.stdout: image: rancher/mirrored-flannelcni-flannel:v0.17.0 2022-03-20T14:52:00.774 INFO:teuthology.orchestra.run.smithi007.stdout: command: 2022-03-20T14:52:00.774 INFO:teuthology.orchestra.run.smithi007.stdout: - /opt/bin/flanneld 2022-03-20T14:52:00.775 INFO:teuthology.orchestra.run.smithi007.stdout: args: 2022-03-20T14:52:00.775 INFO:teuthology.orchestra.run.smithi007.stdout: - --ip-masq 2022-03-20T14:52:00.775 INFO:teuthology.orchestra.run.smithi007.stdout: - --kube-subnet-mgr 2022-03-20T14:52:00.775 INFO:teuthology.orchestra.run.smithi007.stdout: resources: 2022-03-20T14:52:00.775 INFO:teuthology.orchestra.run.smithi007.stdout: requests: 2022-03-20T14:52:00.776 INFO:teuthology.orchestra.run.smithi007.stdout: cpu: "100m" 2022-03-20T14:52:00.776 INFO:teuthology.orchestra.run.smithi007.stdout: memory: "50Mi" 2022-03-20T14:52:00.776 INFO:teuthology.orchestra.run.smithi007.stdout: limits: 2022-03-20T14:52:00.776 INFO:teuthology.orchestra.run.smithi007.stdout: cpu: "100m" 2022-03-20T14:52:00.777 INFO:teuthology.orchestra.run.smithi007.stdout: memory: "50Mi" 2022-03-20T14:52:00.777 INFO:teuthology.orchestra.run.smithi007.stdout: securityContext: 2022-03-20T14:52:00.777 INFO:teuthology.orchestra.run.smithi007.stdout: privileged: false 2022-03-20T14:52:00.777 INFO:teuthology.orchestra.run.smithi007.stdout: capabilities: 2022-03-20T14:52:00.777 INFO:teuthology.orchestra.run.smithi007.stdout: add: ["NET_ADMIN", "NET_RAW"] 2022-03-20T14:52:00.778 INFO:teuthology.orchestra.run.smithi007.stdout: env: 2022-03-20T14:52:00.778 INFO:teuthology.orchestra.run.smithi007.stdout: - name: POD_NAME 2022-03-20T14:52:00.778 INFO:teuthology.orchestra.run.smithi007.stdout: valueFrom: 2022-03-20T14:52:00.778 INFO:teuthology.orchestra.run.smithi007.stdout: fieldRef: 2022-03-20T14:52:00.779 INFO:teuthology.orchestra.run.smithi007.stdout: fieldPath: metadata.name 2022-03-20T14:52:00.779 INFO:teuthology.orchestra.run.smithi007.stdout: - name: POD_NAMESPACE 2022-03-20T14:52:00.779 INFO:teuthology.orchestra.run.smithi007.stdout: valueFrom: 2022-03-20T14:52:00.779 INFO:teuthology.orchestra.run.smithi007.stdout: fieldRef: 2022-03-20T14:52:00.780 INFO:teuthology.orchestra.run.smithi007.stdout: fieldPath: metadata.namespace 2022-03-20T14:52:00.780 INFO:teuthology.orchestra.run.smithi007.stdout: volumeMounts: 2022-03-20T14:52:00.780 INFO:teuthology.orchestra.run.smithi007.stdout: - name: run 2022-03-20T14:52:00.780 INFO:teuthology.orchestra.run.smithi007.stdout: mountPath: /run/flannel 2022-03-20T14:52:00.780 INFO:teuthology.orchestra.run.smithi007.stdout: - name: flannel-cfg 2022-03-20T14:52:00.781 INFO:teuthology.orchestra.run.smithi007.stdout: mountPath: /etc/kube-flannel/ 2022-03-20T14:52:00.781 INFO:teuthology.orchestra.run.smithi007.stdout: - name: xtables-lock 2022-03-20T14:52:00.781 INFO:teuthology.orchestra.run.smithi007.stdout: mountPath: /run/xtables.lock 2022-03-20T14:52:00.781 INFO:teuthology.orchestra.run.smithi007.stdout: volumes: 2022-03-20T14:52:00.782 INFO:teuthology.orchestra.run.smithi007.stdout: - name: run 2022-03-20T14:52:00.782 INFO:teuthology.orchestra.run.smithi007.stdout: hostPath: 2022-03-20T14:52:00.782 INFO:teuthology.orchestra.run.smithi007.stdout: path: /run/flannel 2022-03-20T14:52:00.783 INFO:teuthology.orchestra.run.smithi007.stdout: - name: cni-plugin 2022-03-20T14:52:00.783 INFO:teuthology.orchestra.run.smithi007.stdout: hostPath: 2022-03-20T14:52:00.783 INFO:teuthology.orchestra.run.smithi007.stdout: path: /opt/cni/bin 2022-03-20T14:52:00.783 INFO:teuthology.orchestra.run.smithi007.stdout: - name: cni 2022-03-20T14:52:00.784 INFO:teuthology.orchestra.run.smithi007.stdout: hostPath: 2022-03-20T14:52:00.784 INFO:teuthology.orchestra.run.smithi007.stdout: path: /etc/cni/net.d 2022-03-20T14:52:00.784 INFO:teuthology.orchestra.run.smithi007.stdout: - name: flannel-cfg 2022-03-20T14:52:00.784 INFO:teuthology.orchestra.run.smithi007.stdout: configMap: 2022-03-20T14:52:00.784 INFO:teuthology.orchestra.run.smithi007.stdout: name: kube-flannel-cfg 2022-03-20T14:52:00.785 INFO:teuthology.orchestra.run.smithi007.stdout: - name: xtables-lock 2022-03-20T14:52:00.785 INFO:teuthology.orchestra.run.smithi007.stdout: hostPath: 2022-03-20T14:52:00.785 INFO:teuthology.orchestra.run.smithi007.stdout: path: /run/xtables.lock 2022-03-20T14:52:00.785 INFO:teuthology.orchestra.run.smithi007.stdout: type: FileOrCreate 2022-03-20T14:52:00.812 INFO:tasks.kubeadm:Updating {'kind': 'ConfigMap', 'apiVersion': 'v1', 'metadata': {'name': 'kube-flannel-cfg', 'namespace': 'kube-system', 'labels': {'tier': 'node', 'app': 'flannel'}}, 'data': {'cni-conf.json': '{\n "name": "cbr0",\n "cniVersion": "0.3.1",\n "plugins": [\n {\n "type": "flannel",\n "delegate": {\n "hairpinMode": true,\n "isDefaultGateway": true\n }\n },\n {\n "type": "portmap",\n "capabilities": {\n "portMappings": true\n }\n }\n ]\n}\n', 'net-conf.json': '{\n "Network": "10.244.0.0/16",\n "Backend": {\n "Type": "vxlan"\n }\n}\n'}} 2022-03-20T14:52:00.813 INFO:tasks.kubeadm:Now {'kind': 'ConfigMap', 'apiVersion': 'v1', 'metadata': {'name': 'kube-flannel-cfg', 'namespace': 'kube-system', 'labels': {'tier': 'node', 'app': 'flannel'}}, 'data': {'cni-conf.json': '{\n "name": "cbr0",\n "cniVersion": "0.3.1",\n "plugins": [\n {\n "type": "flannel",\n "delegate": {\n "hairpinMode": true,\n "isDefaultGateway": true\n }\n },\n {\n "type": "portmap",\n "capabilities": {\n "portMappings": true\n }\n }\n ]\n}\n', 'net-conf.json': '{\n "Network": "10.248.48.0/21",\n "Backend": {\n "Type": "vxlan"\n }\n}\n'}} 2022-03-20T14:52:00.826 DEBUG:tasks.kubeadm:Flannel: apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: annotations: apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default name: psp.flannel.unprivileged spec: allowPrivilegeEscalation: false allowedCapabilities: - NET_ADMIN - NET_RAW allowedHostPaths: - pathPrefix: /etc/cni/net.d - pathPrefix: /etc/kube-flannel - pathPrefix: /run/flannel defaultAddCapabilities: [] defaultAllowPrivilegeEscalation: false fsGroup: rule: RunAsAny hostIPC: false hostNetwork: true hostPID: false hostPorts: - max: 65535 min: 0 privileged: false readOnlyRootFilesystem: false requiredDropCapabilities: [] runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny volumes: - configMap - secret - emptyDir - hostPath --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: flannel rules: - apiGroups: - extensions resourceNames: - psp.flannel.unprivileged resources: - podsecuritypolicies verbs: - use - apiGroups: - '' resources: - pods verbs: - get - apiGroups: - '' resources: - nodes verbs: - list - watch - apiGroups: - '' resources: - nodes/status verbs: - patch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: flannel roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: flannel subjects: - kind: ServiceAccount name: flannel namespace: kube-system --- apiVersion: v1 kind: ServiceAccount metadata: name: flannel namespace: kube-system --- apiVersion: v1 data: cni-conf.json: "{\n \"name\": \"cbr0\",\n \"cniVersion\": \"0.3.1\",\n \"plugins\"\ : [\n {\n \"type\": \"flannel\",\n \"delegate\": {\n \"hairpinMode\"\ : true,\n \"isDefaultGateway\": true\n }\n },\n {\n \"\ type\": \"portmap\",\n \"capabilities\": {\n \"portMappings\": true\n\ \ }\n }\n ]\n}\n" net-conf.json: "{\n \"Network\": \"10.248.48.0/21\",\n \"Backend\": {\n \"\ Type\": \"vxlan\"\n }\n}\n" kind: ConfigMap metadata: labels: app: flannel tier: node name: kube-flannel-cfg namespace: kube-system --- apiVersion: apps/v1 kind: DaemonSet metadata: labels: app: flannel tier: node name: kube-flannel-ds namespace: kube-system spec: selector: matchLabels: app: flannel template: metadata: labels: app: flannel tier: node spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/os operator: In values: - linux containers: - args: - --ip-masq - --kube-subnet-mgr command: - /opt/bin/flanneld env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: rancher/mirrored-flannelcni-flannel:v0.17.0 name: kube-flannel resources: limits: cpu: 100m memory: 50Mi requests: cpu: 100m memory: 50Mi securityContext: capabilities: add: - NET_ADMIN - NET_RAW privileged: false volumeMounts: - mountPath: /run/flannel name: run - mountPath: /etc/kube-flannel/ name: flannel-cfg - mountPath: /run/xtables.lock name: xtables-lock hostNetwork: true initContainers: - args: - -f - /flannel - /opt/cni/bin/flannel command: - cp image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.1 name: install-cni-plugin volumeMounts: - mountPath: /opt/cni/bin name: cni-plugin - args: - -f - /etc/kube-flannel/cni-conf.json - /etc/cni/net.d/10-flannel.conflist command: - cp image: rancher/mirrored-flannelcni-flannel:v0.17.0 name: install-cni volumeMounts: - mountPath: /etc/cni/net.d name: cni - mountPath: /etc/kube-flannel/ name: flannel-cfg priorityClassName: system-node-critical serviceAccountName: flannel tolerations: - effect: NoSchedule operator: Exists volumes: - hostPath: path: /run/flannel name: run - hostPath: path: /opt/cni/bin name: cni-plugin - hostPath: path: /etc/cni/net.d name: cni - configMap: name: kube-flannel-cfg name: flannel-cfg - hostPath: path: /run/xtables.lock type: FileOrCreate name: xtables-lock 2022-03-20T14:52:00.827 DEBUG:teuthology.orchestra.run.smithi007:> kubectl apply -f - 2022-03-20T14:52:03.426 INFO:teuthology.orchestra.run.smithi007.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-20T14:52:03.431 INFO:teuthology.orchestra.run.smithi007.stdout:podsecuritypolicy.policy/psp.flannel.unprivileged created 2022-03-20T14:52:03.494 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/flannel created 2022-03-20T14:52:03.507 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/flannel created 2022-03-20T14:52:03.518 INFO:teuthology.orchestra.run.smithi007.stdout:serviceaccount/flannel created 2022-03-20T14:52:03.526 INFO:teuthology.orchestra.run.smithi007.stdout:configmap/kube-flannel-cfg created 2022-03-20T14:52:03.549 INFO:teuthology.orchestra.run.smithi007.stdout:daemonset.apps/kube-flannel-ds created 2022-03-20T14:52:03.553 INFO:tasks.kubeadm:Scanning for scratch devices 2022-03-20T14:52:03.554 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-20T14:52:03.554 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/scratch_devs of=/dev/stdout 2022-03-20T14:52:03.560 INFO:tasks.kubeadm:Scratch devices on smithi007: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-20T14:52:03.561 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-20T14:52:03.583 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records in 2022-03-20T14:52:03.583 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2022-03-20T14:52:03.584 INFO:teuthology.orchestra.run.smithi007.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0108438 s, 967 MB/s 2022-03-20T14:52:03.585 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-20T14:52:03.607 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records in 2022-03-20T14:52:03.607 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2022-03-20T14:52:03.607 INFO:teuthology.orchestra.run.smithi007.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0100755 s, 1.0 GB/s 2022-03-20T14:52:03.608 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-20T14:52:03.632 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records in 2022-03-20T14:52:03.632 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2022-03-20T14:52:03.633 INFO:teuthology.orchestra.run.smithi007.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.014429 s, 727 MB/s 2022-03-20T14:52:03.634 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-20T14:52:03.655 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records in 2022-03-20T14:52:03.656 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2022-03-20T14:52:03.656 INFO:teuthology.orchestra.run.smithi007.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0105569 s, 993 MB/s 2022-03-20T14:52:03.657 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2022-03-20T14:52:03.658 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/scratch_devs of=/dev/stdout 2022-03-20T14:52:03.665 INFO:tasks.kubeadm:Scratch devices on smithi057: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-20T14:52:03.666 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-20T14:52:03.733 INFO:teuthology.orchestra.run.smithi057.stderr:10+0 records in 2022-03-20T14:52:03.733 INFO:teuthology.orchestra.run.smithi057.stderr:10+0 records out 2022-03-20T14:52:03.733 INFO:teuthology.orchestra.run.smithi057.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0143156 s, 732 MB/s 2022-03-20T14:52:03.735 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-20T14:52:03.757 INFO:teuthology.orchestra.run.smithi057.stderr:10+0 records in 2022-03-20T14:52:03.758 INFO:teuthology.orchestra.run.smithi057.stderr:10+0 records out 2022-03-20T14:52:03.758 INFO:teuthology.orchestra.run.smithi057.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113942 s, 920 MB/s 2022-03-20T14:52:03.759 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-20T14:52:03.786 INFO:teuthology.orchestra.run.smithi057.stderr:10+0 records in 2022-03-20T14:52:03.786 INFO:teuthology.orchestra.run.smithi057.stderr:10+0 records out 2022-03-20T14:52:03.786 INFO:teuthology.orchestra.run.smithi057.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0132418 s, 792 MB/s 2022-03-20T14:52:03.788 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-20T14:52:03.860 INFO:teuthology.orchestra.run.smithi057.stderr:10+0 records in 2022-03-20T14:52:03.860 INFO:teuthology.orchestra.run.smithi057.stderr:10+0 records out 2022-03-20T14:52:03.860 INFO:teuthology.orchestra.run.smithi057.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0144193 s, 727 MB/s 2022-03-20T14:52:03.861 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2022-03-20T14:52:03.862 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/scratch_devs of=/dev/stdout 2022-03-20T14:52:03.869 INFO:tasks.kubeadm:Scratch devices on smithi087: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-20T14:52:03.869 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-20T14:52:03.905 INFO:teuthology.orchestra.run.smithi087.stderr:10+0 records in 2022-03-20T14:52:03.905 INFO:teuthology.orchestra.run.smithi087.stderr:10+0 records out 2022-03-20T14:52:03.906 INFO:teuthology.orchestra.run.smithi087.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0229893 s, 456 MB/s 2022-03-20T14:52:03.907 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-20T14:52:03.933 INFO:teuthology.orchestra.run.smithi087.stderr:10+0 records in 2022-03-20T14:52:03.933 INFO:teuthology.orchestra.run.smithi087.stderr:10+0 records out 2022-03-20T14:52:03.933 INFO:teuthology.orchestra.run.smithi087.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0160647 s, 653 MB/s 2022-03-20T14:52:03.934 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-20T14:52:03.962 INFO:teuthology.orchestra.run.smithi087.stderr:10+0 records in 2022-03-20T14:52:03.963 INFO:teuthology.orchestra.run.smithi087.stderr:10+0 records out 2022-03-20T14:52:03.963 INFO:teuthology.orchestra.run.smithi087.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.017421 s, 602 MB/s 2022-03-20T14:52:03.964 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-20T14:52:04.038 INFO:teuthology.orchestra.run.smithi087.stderr:10+0 records in 2022-03-20T14:52:04.038 INFO:teuthology.orchestra.run.smithi087.stderr:10+0 records out 2022-03-20T14:52:04.038 INFO:teuthology.orchestra.run.smithi087.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.018449 s, 568 MB/s 2022-03-20T14:52:04.055 INFO:tasks.kubeadm:Creating PVs + StorageClass 2022-03-20T14:52:04.056 DEBUG:tasks.kubeadm:apiVersion: v1 kind: PersistentVolume metadata: name: smithi007-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi007 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi007-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi007 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi007-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi007 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi007-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi007 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi057-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi057 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi057-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi057 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi057-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi057 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi057-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi057 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi087-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi087 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi087-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi087 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi087-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi087 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi087-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi087 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: scratch provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer 2022-03-20T14:52:04.056 DEBUG:teuthology.orchestra.run.smithi007:> kubectl create -f - 2022-03-20T14:52:04.727 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi007-lv-1 created 2022-03-20T14:52:04.884 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi007-lv-2 created 2022-03-20T14:52:04.888 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi007-lv-3 created 2022-03-20T14:52:04.962 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi007-lv-4 created 2022-03-20T14:52:05.084 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi057-lv-1 created 2022-03-20T14:52:05.127 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi057-lv-2 created 2022-03-20T14:52:05.138 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi057-lv-3 created 2022-03-20T14:52:05.236 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi057-lv-4 created 2022-03-20T14:52:05.240 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi087-lv-1 created 2022-03-20T14:52:05.248 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi087-lv-2 created 2022-03-20T14:52:05.292 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi087-lv-3 created 2022-03-20T14:52:05.328 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi087-lv-4 created 2022-03-20T14:52:05.392 INFO:teuthology.orchestra.run.smithi007.stdout:storageclass.storage.k8s.io/scratch created 2022-03-20T14:52:05.397 DEBUG:teuthology.orchestra.run.smithi007:> kubectl taint node smithi007 node-role.kubernetes.io/master- || true 2022-03-20T14:52:05.470 INFO:teuthology.orchestra.run.smithi007.stdout:node/smithi007 untainted 2022-03-20T14:52:05.473 INFO:tasks.kubeadm:Kubeadm complete, yielding 2022-03-20T14:52:05.473 INFO:teuthology.run_tasks:Running task nvme_loop... 2022-03-20T14:52:05.485 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2022-03-20T14:52:05.485 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-20T14:52:05.486 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/scratch_devs of=/dev/stdout 2022-03-20T14:52:05.497 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-20T14:52:05.498 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-20T14:52:05.511 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-20T14:52:05.511 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-20T14:52:05.511 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 500 Links: 1 2022-03-20T14:52:05.512 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-20T14:52:05.513 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2022-03-20 14:52:03.582631321 +0000 2022-03-20T14:52:05.513 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2022-03-20 14:52:03.582631321 +0000 2022-03-20T14:52:05.514 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2022-03-20 14:52:03.582631321 +0000 2022-03-20T14:52:05.514 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2022-03-20T14:52:05.521 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2022-03-20T14:52:05.521 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2022-03-20T14:52:05.521 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000237946 s, 2.2 MB/s 2022-03-20T14:52:05.523 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-20T14:52:05.585 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-20T14:52:05.586 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-20T14:52:05.586 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 508 Links: 1 2022-03-20T14:52:05.587 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-20T14:52:05.587 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2022-03-20 14:52:03.606631320 +0000 2022-03-20T14:52:05.587 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2022-03-20 14:52:03.606631320 +0000 2022-03-20T14:52:05.588 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2022-03-20 14:52:03.606631320 +0000 2022-03-20T14:52:05.588 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2022-03-20T14:52:05.595 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2022-03-20T14:52:05.596 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2022-03-20T14:52:05.596 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000336161 s, 1.5 MB/s 2022-03-20T14:52:05.598 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-20T14:52:05.655 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-20T14:52:05.656 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-20T14:52:05.658 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 512 Links: 1 2022-03-20T14:52:05.659 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-20T14:52:05.659 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2022-03-20 14:52:03.630631320 +0000 2022-03-20T14:52:05.660 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2022-03-20 14:52:03.630631320 +0000 2022-03-20T14:52:05.660 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2022-03-20 14:52:03.630631320 +0000 2022-03-20T14:52:05.661 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2022-03-20T14:52:05.662 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2022-03-20T14:52:05.663 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2022-03-20T14:52:05.663 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000229783 s, 2.2 MB/s 2022-03-20T14:52:05.665 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-20T14:52:05.719 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-20T14:52:05.720 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-20T14:52:05.720 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 519 Links: 1 2022-03-20T14:52:05.721 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-20T14:52:05.721 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2022-03-20 14:52:03.654631319 +0000 2022-03-20T14:52:05.721 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2022-03-20 14:52:03.654631319 +0000 2022-03-20T14:52:05.722 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2022-03-20 14:52:03.654631319 +0000 2022-03-20T14:52:05.722 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2022-03-20T14:52:05.725 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2022-03-20T14:52:05.725 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2022-03-20T14:52:05.726 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.00024268 s, 2.1 MB/s 2022-03-20T14:52:05.731 DEBUG:teuthology.orchestra.run.smithi007:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-20T14:52:05.916 INFO:teuthology.orchestra.run.smithi007.stdout:loop 2022-03-20T14:52:05.918 INFO:tasks.nvme_loop:Connecting nvme_loop smithi007:/dev/vg_nvme/lv_1... 2022-03-20T14:52:05.918 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-20T14:52:05.940 INFO:teuthology.orchestra.run.smithi007.stdout:1 2022-03-20T14:52:05.964 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/vg_nvme/lv_11 2022-03-20T14:52:05.989 INFO:tasks.nvme_loop:Connecting nvme_loop smithi007:/dev/vg_nvme/lv_2... 2022-03-20T14:52:05.989 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-20T14:52:06.012 INFO:teuthology.orchestra.run.smithi007.stdout:1 2022-03-20T14:52:06.039 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/vg_nvme/lv_21 2022-03-20T14:52:06.062 INFO:tasks.nvme_loop:Connecting nvme_loop smithi007:/dev/vg_nvme/lv_3... 2022-03-20T14:52:06.063 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-20T14:52:06.125 INFO:teuthology.orchestra.run.smithi007.stdout:1 2022-03-20T14:52:06.155 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/vg_nvme/lv_31 2022-03-20T14:52:06.173 INFO:tasks.nvme_loop:Connecting nvme_loop smithi007:/dev/vg_nvme/lv_4... 2022-03-20T14:52:06.173 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-20T14:52:06.194 INFO:teuthology.orchestra.run.smithi007.stdout:1 2022-03-20T14:52:06.223 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/vg_nvme/lv_41 2022-03-20T14:52:06.248 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-20T14:52:06.249 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/scratch_devs of=/dev/stdout 2022-03-20T14:52:06.256 DEBUG:teuthology.orchestra.run.smithi007:> sudo nvme list 2022-03-20T14:52:06.314 INFO:teuthology.orchestra.run.smithi007.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-20T14:52:06.314 INFO:teuthology.orchestra.run.smithi007.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-20T14:52:06.314 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/nvme0n1 PHMB751300GE480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2022-03-20T14:52:06.315 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/nvme1n1 3a3974317138a844 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-20T14:52:06.315 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/nvme2n1 0cc4b7410d4cec06 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-20T14:52:06.315 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/nvme3n1 9eea28d7422bc59c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-20T14:52:06.316 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/nvme4n1 945b496c7d3bd87b Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-20T14:52:06.317 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-20T14:52:06.317 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-20T14:52:06.318 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/scratch_devs 2022-03-20T14:52:06.332 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2022-03-20T14:52:06.332 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/scratch_devs of=/dev/stdout 2022-03-20T14:52:06.338 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-20T14:52:06.339 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-20T14:52:06.390 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-20T14:52:06.390 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-20T14:52:06.390 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 5h/5d Inode: 499 Links: 1 2022-03-20T14:52:06.391 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-20T14:52:06.391 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2022-03-20 14:52:03.738276347 +0000 2022-03-20T14:52:06.391 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2022-03-20 14:52:03.734276347 +0000 2022-03-20T14:52:06.391 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2022-03-20 14:52:03.734276347 +0000 2022-03-20T14:52:06.392 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: - 2022-03-20T14:52:06.396 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2022-03-20T14:52:06.397 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2022-03-20T14:52:06.397 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000423085 s, 1.2 MB/s 2022-03-20T14:52:06.401 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-20T14:52:06.407 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-20T14:52:06.408 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-20T14:52:06.408 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 5h/5d Inode: 505 Links: 1 2022-03-20T14:52:06.408 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-20T14:52:06.408 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2022-03-20 14:52:03.762276348 +0000 2022-03-20T14:52:06.409 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2022-03-20 14:52:03.762276348 +0000 2022-03-20T14:52:06.409 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2022-03-20 14:52:03.762276348 +0000 2022-03-20T14:52:06.409 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: - 2022-03-20T14:52:06.415 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2022-03-20T14:52:06.415 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2022-03-20T14:52:06.415 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000435167 s, 1.2 MB/s 2022-03-20T14:52:06.418 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-20T14:52:06.424 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-20T14:52:06.424 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-20T14:52:06.425 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 5h/5d Inode: 512 Links: 1 2022-03-20T14:52:06.425 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-20T14:52:06.425 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2022-03-20 14:52:03.786276348 +0000 2022-03-20T14:52:06.425 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2022-03-20 14:52:03.786276348 +0000 2022-03-20T14:52:06.426 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2022-03-20 14:52:03.786276348 +0000 2022-03-20T14:52:06.426 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: - 2022-03-20T14:52:06.430 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2022-03-20T14:52:06.431 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2022-03-20T14:52:06.431 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000434859 s, 1.2 MB/s 2022-03-20T14:52:06.434 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-20T14:52:06.440 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-20T14:52:06.441 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-20T14:52:06.441 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 5h/5d Inode: 520 Links: 1 2022-03-20T14:52:06.441 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-20T14:52:06.441 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2022-03-20 14:52:03.866276350 +0000 2022-03-20T14:52:06.442 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2022-03-20 14:52:03.866276350 +0000 2022-03-20T14:52:06.442 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2022-03-20 14:52:03.866276350 +0000 2022-03-20T14:52:06.442 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: - 2022-03-20T14:52:06.449 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2022-03-20T14:52:06.450 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2022-03-20T14:52:06.450 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000434259 s, 1.2 MB/s 2022-03-20T14:52:06.453 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-20T14:52:06.680 INFO:teuthology.orchestra.run.smithi057.stdout:loop 2022-03-20T14:52:06.681 INFO:tasks.nvme_loop:Connecting nvme_loop smithi057:/dev/vg_nvme/lv_1... 2022-03-20T14:52:06.682 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-20T14:52:06.704 INFO:teuthology.orchestra.run.smithi057.stdout:1 2022-03-20T14:52:06.725 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/vg_nvme/lv_11 2022-03-20T14:52:06.746 INFO:tasks.nvme_loop:Connecting nvme_loop smithi057:/dev/vg_nvme/lv_2... 2022-03-20T14:52:06.746 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-20T14:52:06.767 INFO:teuthology.orchestra.run.smithi057.stdout:1 2022-03-20T14:52:06.785 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/vg_nvme/lv_21 2022-03-20T14:52:06.803 INFO:tasks.nvme_loop:Connecting nvme_loop smithi057:/dev/vg_nvme/lv_3... 2022-03-20T14:52:06.803 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-20T14:52:06.865 INFO:teuthology.orchestra.run.smithi057.stdout:1 2022-03-20T14:52:06.887 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/vg_nvme/lv_31 2022-03-20T14:52:06.904 INFO:tasks.nvme_loop:Connecting nvme_loop smithi057:/dev/vg_nvme/lv_4... 2022-03-20T14:52:06.905 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-20T14:52:06.924 INFO:teuthology.orchestra.run.smithi057.stdout:1 2022-03-20T14:52:06.950 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/vg_nvme/lv_41 2022-03-20T14:52:06.973 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2022-03-20T14:52:06.973 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/scratch_devs of=/dev/stdout 2022-03-20T14:52:06.980 DEBUG:teuthology.orchestra.run.smithi057:> sudo nvme list 2022-03-20T14:52:06.992 INFO:teuthology.orchestra.run.smithi057.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-20T14:52:06.992 INFO:teuthology.orchestra.run.smithi057.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-20T14:52:06.992 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/nvme0n1 BTLE7361049H4P0KGN INTEL SSDPEDKE040T7 1 4.00 TB / 4.00 TB 512 B + 0 B QDV1013D 2022-03-20T14:52:06.993 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/nvme1n1 6e5f65c8725db826 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-20T14:52:06.993 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/nvme2n1 cb4ea1f428bfc9f3 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-20T14:52:06.993 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/nvme3n1 2250c8bf019b38b7 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-20T14:52:06.994 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1'] 2022-03-20T14:52:07.995 DEBUG:teuthology.orchestra.run.smithi057:> sudo nvme list 2022-03-20T14:52:08.007 INFO:teuthology.orchestra.run.smithi057.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-20T14:52:08.007 INFO:teuthology.orchestra.run.smithi057.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-20T14:52:08.007 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/nvme0n1 BTLE7361049H4P0KGN INTEL SSDPEDKE040T7 1 4.00 TB / 4.00 TB 512 B + 0 B QDV1013D 2022-03-20T14:52:08.008 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/nvme1n1 6e5f65c8725db826 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-20T14:52:08.008 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/nvme2n1 cb4ea1f428bfc9f3 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-20T14:52:08.008 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/nvme3n1 2250c8bf019b38b7 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-20T14:52:08.008 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/nvme4n1 8b8c8736a9a40e51 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-20T14:52:08.009 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-20T14:52:08.010 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2022-03-20T14:52:08.010 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/scratch_devs 2022-03-20T14:52:08.068 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2022-03-20T14:52:08.069 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/scratch_devs of=/dev/stdout 2022-03-20T14:52:08.074 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-20T14:52:08.074 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-20T14:52:08.080 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-20T14:52:08.080 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-20T14:52:08.081 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 500 Links: 1 2022-03-20T14:52:08.081 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-20T14:52:08.081 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2022-03-20 14:52:03.907002431 +0000 2022-03-20T14:52:08.081 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2022-03-20 14:52:03.907002431 +0000 2022-03-20T14:52:08.082 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2022-03-20 14:52:03.907002431 +0000 2022-03-20T14:52:08.082 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: - 2022-03-20T14:52:08.091 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2022-03-20T14:52:08.091 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2022-03-20T14:52:08.093 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000542864 s, 943 kB/s 2022-03-20T14:52:08.094 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-20T14:52:08.145 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-20T14:52:08.145 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-20T14:52:08.146 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 507 Links: 1 2022-03-20T14:52:08.146 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-20T14:52:08.146 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2022-03-20 14:52:03.939002430 +0000 2022-03-20T14:52:08.146 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2022-03-20 14:52:03.939002430 +0000 2022-03-20T14:52:08.147 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2022-03-20 14:52:03.939002430 +0000 2022-03-20T14:52:08.147 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: - 2022-03-20T14:52:08.154 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2022-03-20T14:52:08.155 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2022-03-20T14:52:08.155 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000589574 s, 868 kB/s 2022-03-20T14:52:08.158 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-20T14:52:08.164 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-20T14:52:08.165 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-20T14:52:08.165 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 512 Links: 1 2022-03-20T14:52:08.165 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-20T14:52:08.165 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2022-03-20 14:52:03.967002429 +0000 2022-03-20T14:52:08.166 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2022-03-20 14:52:03.967002429 +0000 2022-03-20T14:52:08.166 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2022-03-20 14:52:03.967002429 +0000 2022-03-20T14:52:08.166 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: - 2022-03-20T14:52:08.172 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2022-03-20T14:52:08.172 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2022-03-20T14:52:08.172 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.00051616 s, 992 kB/s 2022-03-20T14:52:08.176 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-20T14:52:08.182 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-20T14:52:08.182 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-20T14:52:08.183 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 517 Links: 1 2022-03-20T14:52:08.183 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-20T14:52:08.183 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2022-03-20 14:52:04.043002428 +0000 2022-03-20T14:52:08.183 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2022-03-20 14:52:04.043002428 +0000 2022-03-20T14:52:08.184 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2022-03-20 14:52:04.043002428 +0000 2022-03-20T14:52:08.184 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: - 2022-03-20T14:52:08.188 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2022-03-20T14:52:08.189 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2022-03-20T14:52:08.190 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000548028 s, 934 kB/s 2022-03-20T14:52:08.192 DEBUG:teuthology.orchestra.run.smithi087:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-20T14:52:08.351 INFO:teuthology.orchestra.run.smithi087.stdout:loop 2022-03-20T14:52:08.354 INFO:tasks.nvme_loop:Connecting nvme_loop smithi087:/dev/vg_nvme/lv_1... 2022-03-20T14:52:08.355 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-20T14:52:08.376 INFO:teuthology.orchestra.run.smithi087.stdout:1 2022-03-20T14:52:08.404 INFO:teuthology.orchestra.run.smithi087.stdout:/dev/vg_nvme/lv_11 2022-03-20T14:52:08.425 INFO:tasks.nvme_loop:Connecting nvme_loop smithi087:/dev/vg_nvme/lv_2... 2022-03-20T14:52:08.426 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-20T14:52:08.492 INFO:teuthology.orchestra.run.smithi087.stdout:1 2022-03-20T14:52:08.518 INFO:teuthology.orchestra.run.smithi087.stdout:/dev/vg_nvme/lv_21 2022-03-20T14:52:08.541 INFO:tasks.nvme_loop:Connecting nvme_loop smithi087:/dev/vg_nvme/lv_3... 2022-03-20T14:52:08.542 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-20T14:52:08.567 INFO:teuthology.orchestra.run.smithi087.stdout:1 2022-03-20T14:52:08.589 INFO:teuthology.orchestra.run.smithi087.stdout:/dev/vg_nvme/lv_31 2022-03-20T14:52:08.612 INFO:tasks.nvme_loop:Connecting nvme_loop smithi087:/dev/vg_nvme/lv_4... 2022-03-20T14:52:08.613 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-20T14:52:08.634 INFO:teuthology.orchestra.run.smithi087.stdout:1 2022-03-20T14:52:08.657 INFO:teuthology.orchestra.run.smithi087.stdout:/dev/vg_nvme/lv_41 2022-03-20T14:52:08.679 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2022-03-20T14:52:08.679 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/scratch_devs of=/dev/stdout 2022-03-20T14:52:08.734 DEBUG:teuthology.orchestra.run.smithi087:> sudo nvme list 2022-03-20T14:52:08.745 INFO:teuthology.orchestra.run.smithi087.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-20T14:52:08.745 INFO:teuthology.orchestra.run.smithi087.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-20T14:52:08.746 INFO:teuthology.orchestra.run.smithi087.stdout:/dev/nvme0n1 PHFT620400NQ400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-20T14:52:08.746 INFO:teuthology.orchestra.run.smithi087.stdout:/dev/nvme1n1 defd165489ba0177 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-20T14:52:08.746 INFO:teuthology.orchestra.run.smithi087.stdout:/dev/nvme2n1 855175ec57dbf67f Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-20T14:52:08.747 INFO:teuthology.orchestra.run.smithi087.stdout:/dev/nvme3n1 fcad3dafda6ee890 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-20T14:52:08.747 INFO:teuthology.orchestra.run.smithi087.stdout:/dev/nvme4n1 a901a0b47d86b2bb Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-20T14:52:08.748 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-20T14:52:08.748 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2022-03-20T14:52:08.749 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/scratch_devs 2022-03-20T14:52:08.760 INFO:teuthology.run_tasks:Running task rook... 2022-03-20T14:52:08.837 INFO:tasks.rook:Rook start 2022-03-20T14:52:08.837 INFO:tasks.rook:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'e92561bf8a1bc96ca490936c6f46ac4673e1b26e', 'rook_image': 'rook/ceph:master'} 2022-03-20T14:52:08.838 INFO:tasks.rook:Ceph image is quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e 2022-03-20T14:52:08.838 INFO:tasks.rook:Cloning https://github.com/rook/rook branch master 2022-03-20T14:52:08.839 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf rook && git clone --single-branch --branch master https://github.com/rook/rook rook 2022-03-20T14:52:08.848 INFO:teuthology.orchestra.run.smithi007.stderr:Cloning into 'rook'... 2022-03-20T14:52:11.517 INFO:tasks.rook:/home/teuthworker 2022-03-20T14:52:11.518 DEBUG:teuthology.orchestra.run.smithi007:> PATH=/sbin:/usr/sbin ip addr show 2022-03-20T14:52:11.567 INFO:teuthology.orchestra.run.smithi007.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-03-20T14:52:11.568 INFO:teuthology.orchestra.run.smithi007.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-03-20T14:52:11.568 INFO:teuthology.orchestra.run.smithi007.stdout: inet 127.0.0.1/8 scope host lo 2022-03-20T14:52:11.568 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2022-03-20T14:52:11.568 INFO:teuthology.orchestra.run.smithi007.stdout: inet6 ::1/128 scope host 2022-03-20T14:52:11.569 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2022-03-20T14:52:11.569 INFO:teuthology.orchestra.run.smithi007.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-20T14:52:11.569 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:6b:fd:dc brd ff:ff:ff:ff:ff:ff 2022-03-20T14:52:11.570 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp7s0f0 2022-03-20T14:52:11.570 INFO:teuthology.orchestra.run.smithi007.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-20T14:52:11.571 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:6b:fd:dd brd ff:ff:ff:ff:ff:ff 2022-03-20T14:52:11.571 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp7s0f1 2022-03-20T14:52:11.572 INFO:teuthology.orchestra.run.smithi007.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2022-03-20T14:52:11.572 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:bd:15:ea brd ff:ff:ff:ff:ff:ff 2022-03-20T14:52:11.573 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp1s0f0 2022-03-20T14:52:11.573 INFO:teuthology.orchestra.run.smithi007.stdout: inet 172.21.15.7/20 brd 172.21.15.255 scope global dynamic ens1f0 2022-03-20T14:52:11.573 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft 42835sec preferred_lft 42835sec 2022-03-20T14:52:11.574 INFO:teuthology.orchestra.run.smithi007.stdout: inet6 fe80::ec4:7aff:febd:15ea/64 scope link 2022-03-20T14:52:11.574 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2022-03-20T14:52:11.575 INFO:teuthology.orchestra.run.smithi007.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-20T14:52:11.575 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:bd:15:eb brd ff:ff:ff:ff:ff:ff 2022-03-20T14:52:11.576 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp1s0f1 2022-03-20T14:52:11.576 INFO:teuthology.orchestra.run.smithi007.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2022-03-20T14:52:11.576 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 02:42:39:4a:26:9b brd ff:ff:ff:ff:ff:ff 2022-03-20T14:52:11.577 INFO:teuthology.orchestra.run.smithi007.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2022-03-20T14:52:11.577 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2022-03-20T14:52:11.579 DEBUG:teuthology.orchestra.run.smithi007:> which systemctl 2022-03-20T14:52:11.585 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/systemctl 2022-03-20T14:52:11.585 DEBUG:teuthology.orchestra.run.smithi007:> true 2022-03-20T14:52:11.618 DEBUG:teuthology.orchestra.run.smithi007:> sudo lsb_release -is 2022-03-20T14:52:11.663 INFO:teuthology.orchestra.run.smithi007.stdout:Ubuntu 2022-03-20T14:52:11.667 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-20T14:52:11.668 INFO:tasks.rook:['__class__', '__del__', '__delattr__', '__dir__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_format_size', '_reconnect', '_runner', '_set_iface_and_cidr', '_sftp_get_file', '_sftp_get_size', '_sftp_open_file', '_sftp_put_file', 'chcon', 'chmod', 'connect', 'copy_file', 'ensure_online', 'get_file', 'get_tar', 'get_tar_stream', 'mkdtemp', 'mktemp', 'move_file', 'put_file', 'read_file', 'reconnect', 'remove', 'run', 'sh', 'sh_file', 'sudo_write_file', 'write_file'] 2022-03-20T14:52:11.727 DEBUG:teuthology.orchestra.remote:smithi007:rook/deploy/examples/operator.yaml is 20KB 2022-03-20T14:52:11.739 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-20T14:52:11.739 DEBUG:teuthology.orchestra.run.smithi007:> dd if=rook/deploy/examples/operator.yaml of=/dev/stdout 2022-03-20T14:52:11.748 INFO:tasks.rook:Patching operator to use image rook/ceph:master 2022-03-20T14:52:11.789 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-20T14:52:11.790 DEBUG:teuthology.orchestra.run.smithi007:> dd of=operator.yaml 2022-03-20T14:52:11.798 INFO:tasks.rook:Deploying operator 2022-03-20T14:52:11.808 DEBUG:teuthology.orchestra.remote:smithi007:rook/deploy/examples/operator.yaml is 20KB 2022-03-20T14:52:11.833 DEBUG:teuthology.orchestra.remote:smithi007:rook/deploy/examples/operator.yaml is 20KB 2022-03-20T14:52:11.847 DEBUG:teuthology.orchestra.run.smithi007:> kubectl create -f rook/deploy/examples/crds.yaml -f rook/deploy/examples/common.yaml -f operator.yaml 2022-03-20T14:52:12.142 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created 2022-03-20T14:52:12.160 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephbucketnotifications.ceph.rook.io created 2022-03-20T14:52:12.186 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephbuckettopics.ceph.rook.io created 2022-03-20T14:52:12.313 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephclients.ceph.rook.io created 2022-03-20T14:52:12.574 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created 2022-03-20T14:52:12.595 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemmirrors.ceph.rook.io created 2022-03-20T14:52:12.653 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created 2022-03-20T14:52:12.759 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemsubvolumegroups.ceph.rook.io created 2022-03-20T14:52:12.780 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created 2022-03-20T14:52:12.809 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectrealms.ceph.rook.io created 2022-03-20T14:52:12.948 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created 2022-03-20T14:52:12.995 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created 2022-03-20T14:52:13.110 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzonegroups.ceph.rook.io created 2022-03-20T14:52:13.148 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzones.ceph.rook.io created 2022-03-20T14:52:13.199 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephrbdmirrors.ceph.rook.io created 2022-03-20T14:52:13.313 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/objectbucketclaims.objectbucket.io created 2022-03-20T14:52:13.318 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/objectbuckets.objectbucket.io created 2022-03-20T14:52:13.370 INFO:teuthology.orchestra.run.smithi007.stdout:namespace/rook-ceph created 2022-03-20T14:52:13.406 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-20T14:52:13.505 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-external-provisioner-runner created 2022-03-20T14:52:13.530 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/psp:rook created 2022-03-20T14:52:13.580 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-20T14:52:13.682 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created 2022-03-20T14:52:13.688 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-20T14:52:13.706 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-20T14:52:13.748 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-20T14:52:13.756 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-20T14:52:13.765 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-20T14:52:13.848 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-20T14:52:13.890 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-20T14:52:13.910 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-20T14:52:13.923 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role created 2022-03-20T14:52:14.008 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-20T14:52:14.018 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created 2022-03-20T14:52:14.041 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-20T14:52:14.047 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-20T14:52:14.075 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-20T14:52:14.164 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-20T14:52:14.168 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-20T14:52:14.172 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system-psp created 2022-03-20T14:52:14.181 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-plugin-sa-psp created 2022-03-20T14:52:14.190 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-provisioner-sa-psp created 2022-03-20T14:52:14.197 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-plugin-sa-psp created 2022-03-20T14:52:14.225 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-provisioner-sa-psp created 2022-03-20T14:52:14.247 INFO:teuthology.orchestra.run.smithi007.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-20T14:52:14.248 INFO:teuthology.orchestra.run.smithi007.stdout:podsecuritypolicy.policy/00-rook-privileged created 2022-03-20T14:52:14.259 INFO:teuthology.orchestra.run.smithi007.stdout:role.rbac.authorization.k8s.io/cephfs-external-provisioner-cfg created 2022-03-20T14:52:14.318 INFO:teuthology.orchestra.run.smithi007.stdout:role.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-20T14:52:14.321 INFO:teuthology.orchestra.run.smithi007.stdout:role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created 2022-03-20T14:52:14.335 INFO:teuthology.orchestra.run.smithi007.stdout:role.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-20T14:52:14.377 INFO:teuthology.orchestra.run.smithi007.stdout:role.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-20T14:52:14.402 INFO:teuthology.orchestra.run.smithi007.stdout:role.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-20T14:52:14.406 INFO:teuthology.orchestra.run.smithi007.stdout:role.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-20T14:52:14.481 INFO:teuthology.orchestra.run.smithi007.stdout:role.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-20T14:52:14.484 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role-cfg created 2022-03-20T14:52:14.519 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin-role-cfg created 2022-03-20T14:52:14.531 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created 2022-03-20T14:52:14.539 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-20T14:52:14.547 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-20T14:52:14.556 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter-psp created 2022-03-20T14:52:14.564 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-default-psp created 2022-03-20T14:52:14.636 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-20T14:52:14.639 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-psp created 2022-03-20T14:52:14.665 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-20T14:52:14.696 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-20T14:52:14.721 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd-psp created 2022-03-20T14:52:14.798 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-20T14:52:14.800 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-20T14:52:14.824 INFO:teuthology.orchestra.run.smithi007.stdout:serviceaccount/rook-ceph-cmd-reporter created 2022-03-20T14:52:14.848 INFO:teuthology.orchestra.run.smithi007.stdout:serviceaccount/rook-ceph-mgr created 2022-03-20T14:52:14.855 INFO:teuthology.orchestra.run.smithi007.stdout:serviceaccount/rook-ceph-osd created 2022-03-20T14:52:14.888 INFO:teuthology.orchestra.run.smithi007.stdout:serviceaccount/rook-ceph-purge-osd created 2022-03-20T14:52:14.897 INFO:teuthology.orchestra.run.smithi007.stdout:serviceaccount/rook-ceph-system created 2022-03-20T14:52:14.993 INFO:teuthology.orchestra.run.smithi007.stdout:serviceaccount/rook-csi-cephfs-plugin-sa created 2022-03-20T14:52:14.998 INFO:teuthology.orchestra.run.smithi007.stdout:serviceaccount/rook-csi-cephfs-provisioner-sa created 2022-03-20T14:52:15.059 INFO:teuthology.orchestra.run.smithi007.stdout:serviceaccount/rook-csi-rbd-plugin-sa created 2022-03-20T14:52:15.089 INFO:teuthology.orchestra.run.smithi007.stdout:serviceaccount/rook-csi-rbd-provisioner-sa created 2022-03-20T14:52:15.182 INFO:teuthology.orchestra.run.smithi007.stdout:configmap/rook-ceph-operator-config created 2022-03-20T14:52:15.207 INFO:teuthology.orchestra.run.smithi007.stdout:deployment.apps/rook-ceph-operator created 2022-03-20T14:52:15.211 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-20T14:52:15.531 INFO:teuthology.orchestra.run.smithi007.stderr:No resources found in rook-ceph namespace. 2022-03-20T14:52:25.534 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-20T14:52:26.418 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:52:26.419 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-operator-75dd789779-lct75 0/1 ContainerCreating 0 11s 2022-03-20T14:52:36.421 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-20T14:52:36.487 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:52:36.488 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-operator-75dd789779-lct75 0/1 ContainerCreating 0 21s 2022-03-20T14:52:46.492 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-20T14:52:46.558 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:52:46.558 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-operator-75dd789779-lct75 0/1 ContainerCreating 0 31s 2022-03-20T14:52:56.561 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-20T14:52:56.627 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:52:56.628 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-operator-75dd789779-lct75 1/1 Running 0 41s 2022-03-20T14:52:56.630 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph logs -f rook-ceph-operator-75dd789779-lct75 2022-03-20T14:52:56.639 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-20T14:52:56.639 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/scratch_devs of=/dev/stdout 2022-03-20T14:52:56.646 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2022-03-20T14:52:56.647 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/scratch_devs of=/dev/stdout 2022-03-20T14:52:56.655 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2022-03-20T14:52:56.716 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/scratch_devs of=/dev/stdout 2022-03-20T14:52:56.733 INFO:tasks.rook: override: [mgr] debug mgr = 20 2022-03-20T14:52:56.733 INFO:tasks.rook: override: [mgr] debug ms = 1 2022-03-20T14:52:56.734 INFO:tasks.rook: override: [mon] debug mon = 20 2022-03-20T14:52:56.734 INFO:tasks.rook: override: [mon] debug ms = 1 2022-03-20T14:52:56.734 INFO:tasks.rook: override: [mon] debug paxos = 20 2022-03-20T14:52:56.735 INFO:tasks.rook: override: [osd] debug ms = 1 2022-03-20T14:52:56.735 INFO:tasks.rook: override: [osd] debug osd = 20 2022-03-20T14:52:56.735 INFO:tasks.rook:Config: b"[global]\n\nlog to file = true\n\nmon clock drift allowed = 1.000\n\n# replicate across OSDs, not hosts\nosd crush chooseleaf type = 0\n\n# enable some debugging\nauth debug = true\nms die on old message = true\nms die on bug = true\ndebug asserts on shutdown = true\n\n\n[osd]\n# debugging\nosd debug shutdown = true\nosd debug op order = true\nosd debug verify stray on activate = true\nosd debug pg log writeout = true\nosd debug verify cached snaps = true\nosd debug verify missing on start = true\nosd debug misdirected ops = true\nosd op queue = debug_random\nosd op queue cut off = debug_random\nosd shutdown pgref assert = true\nbdev debug aio = true\nosd sloppy crc = true\ndebug ms = 1\ndebug osd = 20\n\n\n[mon]\n# rotate auth tickets quickly to exercise renewal paths\nauth mon ticket ttl = 660# 11m\nauth service ticket ttl = 240# 4m\n\n# don't complain about global id reclaim\nmon_warn_on_insecure_global_id_reclaim = false\nmon_warn_on_insecure_global_id_reclaim_allowed = false\ndebug mon = 20\ndebug ms = 1\ndebug paxos = 20\n[mgr]\ndebug mgr = 20\ndebug ms = 1\n" 2022-03-20T14:52:56.739 DEBUG:teuthology.orchestra.run.smithi007:> kubectl create -f - 2022-03-20T14:52:56.741 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.252431 I | rookcmd: starting Rook v1.8.0-alpha.0.441.g2b2469cf1 with arguments '/usr/local/bin/rook ceph operator' 2022-03-20T14:52:56.741 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.252561 I | rookcmd: flag values: --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-level=INFO, --operator-image=, --service-account= 2022-03-20T14:52:56.742 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.252564 I | cephcmd: starting Rook-Ceph operator 2022-03-20T14:52:56.742 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.405917 I | cephcmd: base ceph version inside the rook operator image is "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)" 2022-03-20T14:52:56.742 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.412950 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-20T14:52:56.742 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.412975 I | operator: watching all namespaces for Ceph CRs 2022-03-20T14:52:56.743 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.413094 I | operator: setting up schemes 2022-03-20T14:52:56.743 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.418807 I | operator: setting up the controller-runtime manager 2022-03-20T14:52:56.743 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.627937 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-20T14:52:56.743 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.631115 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-20T14:52:56.743 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.631166 I | ceph-cluster-controller: successfully started 2022-03-20T14:52:56.744 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.631262 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-20T14:52:56.744 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.631292 I | ceph-crashcollector-controller: successfully started 2022-03-20T14:52:56.744 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.631321 I | ceph-block-pool-controller: successfully started 2022-03-20T14:52:56.744 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.631355 I | ceph-object-store-user-controller: successfully started 2022-03-20T14:52:56.744 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.631388 I | ceph-object-realm-controller: successfully started 2022-03-20T14:52:56.745 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.631408 I | ceph-object-zonegroup-controller: successfully started 2022-03-20T14:52:56.745 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.631424 I | ceph-object-zone-controller: successfully started 2022-03-20T14:52:56.747 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.631589 I | ceph-object-controller: successfully started 2022-03-20T14:52:56.748 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.631640 I | ceph-file-controller: successfully started 2022-03-20T14:52:56.748 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.631695 I | ceph-nfs-controller: successfully started 2022-03-20T14:52:56.748 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.631764 I | ceph-rbd-mirror-controller: successfully started 2022-03-20T14:52:56.748 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.631832 I | ceph-client-controller: successfully started 2022-03-20T14:52:56.749 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.631874 I | ceph-filesystem-mirror-controller: successfully started 2022-03-20T14:52:56.749 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.631922 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-20T14:52:56.749 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.631955 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-20T14:52:56.749 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.631988 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-20T14:52:56.750 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.632016 I | ceph-bucket-topic: successfully started 2022-03-20T14:52:56.750 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.632042 I | ceph-bucket-notification: successfully started 2022-03-20T14:52:56.750 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.632069 I | ceph-bucket-notification: successfully started 2022-03-20T14:52:56.750 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.632095 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-20T14:52:56.751 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.633619 I | operator: starting the controller-runtime manager 2022-03-20T14:52:56.751 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.736068 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-20T14:52:56.751 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.736099 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-20T14:52:56.751 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.736110 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-20T14:52:56.751 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.741348 I | ceph-csi: CSI Ceph RBD driver disabled 2022-03-20T14:52:56.752 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.741379 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists 2022-03-20T14:52:56.752 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.760739 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-20T14:52:56.752 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.793787 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists 2022-03-20T14:52:56.752 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.856502 I | ceph-csi: successfully removed CSI Ceph RBD driver 2022-03-20T14:52:56.753 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.856527 I | ceph-csi: CSI CephFS driver disabled 2022-03-20T14:52:56.753 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.856542 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists 2022-03-20T14:52:56.753 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.858945 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists 2022-03-20T14:52:56.753 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:52.865541 I | ceph-csi: successfully removed CSI CephFS driver 2022-03-20T14:52:57.630 INFO:teuthology.orchestra.run.smithi007.stdout:configmap/rook-config-override created 2022-03-20T14:52:57.636 INFO:tasks.rook:Cluster: apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: allowUnsupported: true image: quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e dataDirHostPath: /var/lib/rook mgr: count: 1 modules: - enabled: true name: rook mon: allowMultiplePerNode: true count: 3 skipUpgradeChecks: true 2022-03-20T14:52:57.637 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-20T14:52:57.637 DEBUG:teuthology.orchestra.run.smithi007:> dd of=cluster.yaml 2022-03-20T14:52:57.643 DEBUG:teuthology.orchestra.run.smithi007:> kubectl create -f cluster.yaml 2022-03-20T14:52:59.890 INFO:teuthology.orchestra.run.smithi007.stdout:cephcluster.ceph.rook.io/rook-ceph created 2022-03-20T14:52:59.897 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:59.897961 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "rook-ceph" 2022-03-20T14:52:59.902 DEBUG:teuthology.orchestra.remote:smithi007:rook/deploy/examples/operator.yaml is 20KB 2022-03-20T14:52:59.913 DEBUG:teuthology.orchestra.run.smithi007:> kubectl create -f rook/deploy/examples/toolbox.yaml 2022-03-20T14:52:59.944 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:59.945331 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-20T14:52:59.949 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:59.947440 I | ceph-cluster-controller: clusterInfo not yet found, must be a new cluster. 2022-03-20T14:52:59.962 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:59.962749 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-20T14:52:59.972 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:52:59.971083 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e... 2022-03-20T14:53:00.002 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:53:00.002058 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-20T14:53:00.003 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:53:00.002086 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-20T14:53:00.003 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:53:00.002095 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-20T14:53:00.003 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:53:00.002105 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-20T14:53:00.003 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:53:00.002113 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-20T14:53:00.004 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:53:00.002123 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-20T14:53:00.004 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:53:00.002134 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-20T14:53:00.004 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:53:00.002144 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-20T14:53:00.004 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:53:00.002154 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-20T14:53:00.004 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:53:00.002164 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-20T14:53:00.005 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:53:00.002172 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-20T14:53:00.005 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:53:00.002183 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-20T14:53:00.005 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:53:00.002193 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-20T14:53:00.005 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:53:00.002201 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-20T14:53:00.006 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:53:00.002209 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-20T14:53:00.006 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:53:00.002220 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-20T14:53:00.006 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:53:00.002301 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-20T14:53:00.006 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:53:00.002325 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-20T14:53:00.276 INFO:teuthology.orchestra.run.smithi007.stdout:deployment.apps/rook-ceph-tools created 2022-03-20T14:53:00.282 INFO:tasks.rook:Waiting for tools container to start 2022-03-20T14:53:00.283 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:53:00.453 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:53:00.453 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-64sd5 0/1 ContainerCreating 0 0s 2022-03-20T14:53:05.457 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:53:05.524 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:53:05.525 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-64sd5 0/1 ContainerCreating 0 5s 2022-03-20T14:53:10.527 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:53:10.594 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:53:10.595 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-64sd5 0/1 ContainerCreating 0 10s 2022-03-20T14:53:15.597 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:53:15.667 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:53:15.667 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-64sd5 0/1 ContainerCreating 0 15s 2022-03-20T14:53:20.669 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:53:20.735 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:53:20.736 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-64sd5 0/1 ContainerCreating 0 20s 2022-03-20T14:53:25.739 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:53:25.814 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:53:25.814 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-64sd5 0/1 ContainerCreating 0 25s 2022-03-20T14:53:30.817 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:53:30.884 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:53:30.885 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-64sd5 0/1 ContainerCreating 0 30s 2022-03-20T14:53:35.887 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:53:35.962 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:53:35.963 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-64sd5 0/1 ContainerCreating 0 35s 2022-03-20T14:53:40.966 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:53:41.033 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:53:41.033 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-64sd5 0/1 ContainerCreating 0 41s 2022-03-20T14:53:46.036 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:53:46.106 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:53:46.106 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-64sd5 0/1 ContainerCreating 0 46s 2022-03-20T14:53:51.109 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:53:51.181 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:53:51.182 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-64sd5 0/1 ContainerCreating 0 51s 2022-03-20T14:53:56.184 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:53:56.250 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:53:56.250 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-64sd5 0/1 ContainerCreating 0 56s 2022-03-20T14:54:01.253 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:54:01.320 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:54:01.320 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-64sd5 0/1 ContainerCreating 0 61s 2022-03-20T14:54:06.323 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:54:06.389 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:54:06.390 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-64sd5 0/1 ContainerCreating 0 66s 2022-03-20T14:54:08.456 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:08.456040 I | ceph-spec: detected ceph image version: "17.0.0-11187 quincy" 2022-03-20T14:54:08.456 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:08.456073 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-20T14:54:08.465 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:08.465730 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11187 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e" 2022-03-20T14:54:08.539 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:08.539194 I | op-mon: start running mons 2022-03-20T14:54:08.627 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:08.627089 I | op-mon: creating mon secrets for a new cluster 2022-03-20T14:54:08.648 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:08.648338 I | op-mon: existing maxMonID not found or failed to load. configmaps "rook-ceph-mon-endpoints" not found 2022-03-20T14:54:08.656 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:08.651789 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[]}] data: mapping:{"node":{}} maxMonId:-1] 2022-03-20T14:54:09.384 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:09.383576 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T14:54:09.384 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:09.383853 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T14:54:10.784 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:10.783412 I | op-mon: parsing mon endpoints: 2022-03-20T14:54:10.784 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:10.783442 W | op-mon: ignoring invalid monitor 2022-03-20T14:54:10.785 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:10.783483 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-20T14:54:10.785 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:10.783495 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-20T14:54:10.794 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:10.786216 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-20T14:54:10.794 INFO:tasks.rook.operator.smithi007.stdout:I0320 14:54:10.786318 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-20T14:54:11.385 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:11.384643 I | op-mon: targeting the mon count 3 2022-03-20T14:54:11.392 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:54:11.401 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:11.401035 I | op-mon: created canary deployment rook-ceph-mon-a-canary 2022-03-20T14:54:11.467 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:11.466800 I | op-mon: created canary deployment rook-ceph-mon-b-canary 2022-03-20T14:54:11.469 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:54:11.469 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-64sd5 0/1 ContainerCreating 0 71s 2022-03-20T14:54:11.506 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:11.504317 I | op-mon: created canary deployment rook-ceph-mon-c-canary 2022-03-20T14:54:12.185 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:12.185340 I | op-mon: canary monitor deployment rook-ceph-mon-a-canary scheduled to smithi087 2022-03-20T14:54:12.186 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:12.185376 I | op-mon: mon a assigned to node smithi087 2022-03-20T14:54:12.383 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:12.383361 I | op-mon: canary monitor deployment rook-ceph-mon-b-canary scheduled to smithi057 2022-03-20T14:54:12.384 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:12.383391 I | op-mon: mon b assigned to node smithi057 2022-03-20T14:54:12.584 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:12.583954 I | op-mon: canary monitor deployment rook-ceph-mon-c-canary scheduled to smithi007 2022-03-20T14:54:12.584 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:12.583994 I | op-mon: mon c assigned to node smithi007 2022-03-20T14:54:12.590 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:12.589716 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-a-canary" 2022-03-20T14:54:12.594 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:12.594322 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-b-canary" 2022-03-20T14:54:12.684 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:12.683896 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-c-canary" 2022-03-20T14:54:12.894 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:12.893528 I | op-mon: creating mon a 2022-03-20T14:54:12.973 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:12.973510 I | op-mon: mon "a" ip is 10.98.115.144 2022-03-20T14:54:13.398 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:13.396267 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.98.115.144:6789"]}] data:a=10.98.115.144:6789 mapping:{"node":{"a":{"Name":"smithi087","Hostname":"smithi087","Address":"172.21.15.87"},"b":{"Name":"smithi057","Hostname":"smithi057","Address":"172.21.15.57"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:-1] 2022-03-20T14:54:13.398 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:13.396397 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-20T14:54:13.398 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:13.396513 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-20T14:54:13.983 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:13.983342 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T14:54:13.988 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:13.983666 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T14:54:14.396 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:14.395433 I | op-mon: 0 of 1 expected mons are ready. creating or updating deployments without checking quorum in attempt to achieve a healthy mon cluster 2022-03-20T14:54:14.672 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:14.672261 I | op-mon: updating maxMonID from -1 to 0 after committing mon "a" 2022-03-20T14:54:15.384 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:15.383595 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.98.115.144:6789"]}] data:a=10.98.115.144:6789 mapping:{"node":{"a":{"Name":"smithi087","Hostname":"smithi087","Address":"172.21.15.87"},"b":{"Name":"smithi057","Hostname":"smithi057","Address":"172.21.15.57"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:0] 2022-03-20T14:54:15.384 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:15.383631 I | op-mon: waiting for mon quorum with [a] 2022-03-20T14:54:15.631 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:15.631692 I | op-mon: mons running: [a] 2022-03-20T14:54:16.473 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:54:16.546 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:54:16.547 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-64sd5 0/1 ContainerCreating 0 76s 2022-03-20T14:54:21.551 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:54:21.621 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:54:21.621 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-64sd5 0/1 ContainerCreating 0 81s 2022-03-20T14:54:26.627 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:54:26.693 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:54:26.693 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-64sd5 0/1 ContainerCreating 0 86s 2022-03-20T14:54:31.699 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:54:31.775 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:54:31.776 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-64sd5 0/1 ContainerCreating 0 91s 2022-03-20T14:54:36.037 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:36.037668 I | op-mon: mons running: [a] 2022-03-20T14:54:36.284 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:36.284536 I | op-mon: Monitors in quorum: [a] 2022-03-20T14:54:36.285 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:36.284572 I | op-mon: mons created: 1 2022-03-20T14:54:36.603 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:36.602946 I | op-mon: waiting for mon quorum with [a] 2022-03-20T14:54:36.778 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:54:36.936 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:54:36.937 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-64sd5 0/1 ContainerCreating 0 96s 2022-03-20T14:54:36.941 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:36.937351 I | op-mon: mons running: [a] 2022-03-20T14:54:37.173 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:37.173772 I | op-mon: Monitors in quorum: [a] 2022-03-20T14:54:37.174 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:37.173816 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T14:54:38.738 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:38.738141 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T14:54:38.738 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:38.738180 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T14:54:39.450 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:39.449909 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T14:54:39.450 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:39.449949 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T14:54:40.413 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:40.413561 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T14:54:40.414 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:40.413614 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-20T14:54:41.884 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:41.884236 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-20T14:54:41.884 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:41.884291 I | op-config: deleting "log file" option from the mon configuration database 2022-03-20T14:54:41.939 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:54:42.002 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:54:42.003 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-64sd5 0/1 ContainerCreating 0 102s 2022-03-20T14:54:42.250 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:42.250298 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-20T14:54:42.250 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:42.250339 I | op-mon: creating mon b 2022-03-20T14:54:43.918 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:43.919055 I | op-mon: mon "a" ip is 10.98.115.144 2022-03-20T14:54:44.410 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:44.410267 I | op-mon: mon "b" ip is 10.109.19.50 2022-03-20T14:54:44.900 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:44.900753 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.98.115.144:6789","10.109.19.50:6789"]}] data:a=10.98.115.144:6789,b=10.109.19.50:6789 mapping:{"node":{"a":{"Name":"smithi087","Hostname":"smithi087","Address":"172.21.15.87"},"b":{"Name":"smithi057","Hostname":"smithi057","Address":"172.21.15.57"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:0] 2022-03-20T14:54:45.032 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:45.032852 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T14:54:45.032 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:45.033125 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T14:54:45.211 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:45.211473 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-20T14:54:45.269 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:45.269715 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-20T14:54:45.269 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:45.269747 I | op-mon: waiting for mon quorum with [a b] 2022-03-20T14:54:45.276 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:45.277448 I | op-mon: mons running: [a b] 2022-03-20T14:54:45.954 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:45.954230 I | op-mon: Monitors in quorum: [a] 2022-03-20T14:54:46.033 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:45.997871 I | op-mon: updating maxMonID from 0 to 1 after committing mon "b" 2022-03-20T14:54:46.149 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:46.149594 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.98.115.144:6789","10.109.19.50:6789"]}] data:a=10.98.115.144:6789,b=10.109.19.50:6789 mapping:{"node":{"a":{"Name":"smithi087","Hostname":"smithi087","Address":"172.21.15.87"},"b":{"Name":"smithi057","Hostname":"smithi057","Address":"172.21.15.57"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:1] 2022-03-20T14:54:46.149 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:46.149626 I | op-mon: waiting for mon quorum with [a b] 2022-03-20T14:54:46.159 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:46.159576 I | op-mon: mons running: [a b] 2022-03-20T14:54:46.510 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:46.510782 I | op-mon: Monitors in quorum: [a] 2022-03-20T14:54:46.511 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:46.510812 I | op-mon: mons created: 2 2022-03-20T14:54:46.911 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:46.911953 I | op-mon: waiting for mon quorum with [a b] 2022-03-20T14:54:46.953 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:46.954296 I | op-mon: mons running: [a b] 2022-03-20T14:54:47.006 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:54:47.072 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:54:47.073 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-64sd5 0/1 ContainerCreating 0 107s 2022-03-20T14:54:47.170 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:47.170862 W | op-mon: failed to find initial monitor b in mon map 2022-03-20T14:54:47.171 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:47.170880 W | op-mon: monitor b is not in quorum list 2022-03-20T14:54:48.885 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:48.885339 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-20T14:54:48.886 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:48.885385 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-20T14:54:48.886 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:48.885397 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-20T14:54:48.886 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:48.885421 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-20T14:54:48.886 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:48.885430 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-20T14:54:48.887 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:48.885433 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-20T14:54:48.887 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:48.885436 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-20T14:54:48.887 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:48.885439 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-20T14:54:48.887 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:48.885442 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-20T14:54:48.889 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:48.885445 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-20T14:54:48.889 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:48.885448 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-20T14:54:48.889 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:48.885451 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-20T14:54:48.890 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:48.885454 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-20T14:54:48.890 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:48.885473 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-20T14:54:48.891 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:48.885479 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-20T14:54:48.891 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:48.885484 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-20T14:54:48.891 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:48.885490 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-20T14:54:48.892 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:48.885494 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-20T14:54:48.892 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:48.885499 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-20T14:54:48.893 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:48.885520 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-20T14:54:48.893 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:48.885526 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-20T14:54:48.894 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:48.885530 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-20T14:54:48.894 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:48.885542 I | ceph-csi: Kubernetes version is 1.23 2022-03-20T14:54:48.894 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:48.885549 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-20T14:54:48.895 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:48.885554 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-20T14:54:48.896 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:48.894779 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-20T14:54:48.902 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:48.902184 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-20T14:54:48.903 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:48.902206 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-20T14:54:48.903 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:48.902210 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-20T14:54:48.904 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:48.902213 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-20T14:54:48.904 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:48.902216 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-20T14:54:48.904 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:48.902219 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-20T14:54:48.905 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:48.902222 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-20T14:54:48.961 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:48.961221 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-20T14:54:48.961 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:48.961249 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-20T14:54:48.962 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:48.961259 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-20T14:54:49.057 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:49.058045 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-20T14:54:49.128 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:49.122322 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-20T14:54:49.128 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:49.122364 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-20T14:54:49.129 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:49.122376 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-20T14:54:49.246 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:49.246191 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-20T14:54:49.246 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:49.246213 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-20T14:54:49.247 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:49.246218 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-20T14:54:49.288 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:49.288356 I | ceph-csi: successfully started CSI CephFS driver 2022-03-20T14:54:49.437 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:49.436256 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-20T14:54:50.044 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:50.044598 I | ceph-csi: CSIDriver object created for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-20T14:54:50.044 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:50.044634 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-20T14:54:50.226 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:50.225639 I | ceph-csi: CSIDriver object created for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-20T14:54:52.075 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:54:52.144 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:54:52.144 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-64sd5 0/1 ContainerCreating 0 112s 2022-03-20T14:54:52.187 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:52.187997 I | op-mon: mon b is not yet running 2022-03-20T14:54:52.187 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:52.188021 I | op-mon: mons running: [a] 2022-03-20T14:54:57.147 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:54:57.219 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:57.220284 I | op-mon: mon b is not yet running 2022-03-20T14:54:57.219 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:54:57.220312 I | op-mon: mons running: [a] 2022-03-20T14:54:57.227 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:54:57.228 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-64sd5 0/1 ContainerCreating 0 117s 2022-03-20T14:55:02.230 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:55:02.234 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:02.234538 I | op-mon: mon b is not yet running 2022-03-20T14:55:02.234 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:02.234573 I | op-mon: mons running: [a] 2022-03-20T14:55:02.304 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:55:02.304 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-64sd5 0/1 ContainerCreating 0 2m2s 2022-03-20T14:55:07.262 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:07.262695 I | op-mon: mon b is not yet running 2022-03-20T14:55:07.262 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:07.262731 I | op-mon: mons running: [a] 2022-03-20T14:55:07.307 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:55:07.383 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:55:07.384 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-64sd5 0/1 ContainerCreating 0 2m7s 2022-03-20T14:55:12.291 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:12.291869 I | op-mon: mon b is not yet running 2022-03-20T14:55:12.291 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:12.291906 I | op-mon: mons running: [a] 2022-03-20T14:55:12.386 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:55:12.455 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:55:12.455 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-64sd5 0/1 ContainerCreating 0 2m12s 2022-03-20T14:55:17.459 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:55:18.143 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:55:18.143 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-64sd5 0/1 ContainerCreating 0 2m18s 2022-03-20T14:55:18.512 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:18.513360 I | op-mon: mon b is not yet running 2022-03-20T14:55:18.513 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:18.513402 I | op-mon: mons running: [a] 2022-03-20T14:55:23.146 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:55:23.217 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:55:23.218 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-64sd5 0/1 ContainerCreating 0 2m23s 2022-03-20T14:55:23.552 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:23.552914 I | op-mon: mon b is not yet running 2022-03-20T14:55:23.552 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:23.552953 I | op-mon: mons running: [a] 2022-03-20T14:55:28.220 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-20T14:55:28.288 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-20T14:55:28.289 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-64sd5 1/1 Running 0 2m28s 2022-03-20T14:55:28.291 INFO:tasks.rook:Waiting for mgr/rook orchestrator to be available 2022-03-20T14:55:28.291 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch status -f json 2022-03-20T14:55:28.586 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:28.587636 I | op-mon: mons running: [a b] 2022-03-20T14:55:31.923 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:31.923115 I | op-mon: Monitors in quorum: [a b] 2022-03-20T14:55:31.924 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:31.923157 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T14:55:31.947 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T14:55:31.947 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2022-03-20T14:55:31.948 INFO:teuthology.orchestra.run.smithi007.stderr:pg stat 2022-03-20T14:55:31.949 INFO:teuthology.orchestra.run.smithi007.stderr:pg getmap 2022-03-20T14:55:31.950 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-20T14:55:31.950 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-20T14:55:31.950 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_pools_json 2022-03-20T14:55:31.951 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-pool [...] 2022-03-20T14:55:31.951 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-primary [] [...] 2022-03-20T14:55:31.952 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-osd [] [...] 2022-03-20T14:55:31.952 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls [] [...] 2022-03-20T14:55:31.952 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-20T14:55:31.953 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2022-03-20T14:55:31.954 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-20T14:55:31.955 INFO:teuthology.orchestra.run.smithi007.stderr:command terminated with exit code 22 2022-03-20T14:55:32.147 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:32.148438 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T14:55:32.148 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:32.148476 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T14:55:32.389 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:32.390149 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T14:55:32.389 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:32.390185 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T14:55:32.636 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:32.637235 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T14:55:32.636 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:32.637263 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-20T14:55:32.905 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:32.906551 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-20T14:55:32.905 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:32.906605 I | op-config: deleting "log file" option from the mon configuration database 2022-03-20T14:55:33.168 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:33.169388 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-20T14:55:33.169 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:33.169420 I | op-mon: creating mon c 2022-03-20T14:55:33.296 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:33.297904 I | op-mon: mon "a" ip is 10.98.115.144 2022-03-20T14:55:33.381 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:33.382391 I | op-mon: mon "b" ip is 10.109.19.50 2022-03-20T14:55:33.497 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:33.496411 I | op-mon: mon "c" ip is 10.106.65.43 2022-03-20T14:55:33.573 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:33.574453 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.106.65.43:6789","10.98.115.144:6789","10.109.19.50:6789"]}] data:a=10.98.115.144:6789,b=10.109.19.50:6789,c=10.106.65.43:6789 mapping:{"node":{"a":{"Name":"smithi087","Hostname":"smithi087","Address":"172.21.15.87"},"b":{"Name":"smithi057","Hostname":"smithi057","Address":"172.21.15.57"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:1] 2022-03-20T14:55:34.179 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:34.179875 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T14:55:34.179 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:34.180082 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T14:55:34.579 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:34.581046 I | op-mon: 2 of 3 expected mon deployments exist. creating new deployment(s). 2022-03-20T14:55:34.585 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:34.586497 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-20T14:55:34.598 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:34.600188 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-20T14:55:34.641 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:34.642649 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-20T14:55:34.651 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:34.652925 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-20T14:55:34.800 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:34.801646 I | op-mon: updating maxMonID from 1 to 2 after committing mon "c" 2022-03-20T14:55:35.573 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:35.574442 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.98.115.144:6789","10.109.19.50:6789","10.106.65.43:6789"]}] data:a=10.98.115.144:6789,b=10.109.19.50:6789,c=10.106.65.43:6789 mapping:{"node":{"a":{"Name":"smithi087","Hostname":"smithi087","Address":"172.21.15.87"},"b":{"Name":"smithi057","Hostname":"smithi057","Address":"172.21.15.57"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:2] 2022-03-20T14:55:35.573 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:35.574474 I | op-mon: waiting for mon quorum with [a b c] 2022-03-20T14:55:36.208 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:36.209984 I | op-mon: mon c is not yet running 2022-03-20T14:55:36.209 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:36.210017 I | op-mon: mons running: [a b] 2022-03-20T14:55:36.467 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:36.468353 I | op-mon: Monitors in quorum: [a b] 2022-03-20T14:55:36.467 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:36.468378 I | op-mon: mons created: 3 2022-03-20T14:55:36.724 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:36.725502 I | op-mon: waiting for mon quorum with [a b c] 2022-03-20T14:55:37.937 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:37.938966 I | op-mon: mon c is not yet running 2022-03-20T14:55:37.938 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:37.939018 I | op-mon: mons running: [a b] 2022-03-20T14:55:41.956 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch status -f json 2022-03-20T14:55:42.915 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T14:55:42.915 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2022-03-20T14:55:42.916 INFO:teuthology.orchestra.run.smithi007.stderr:pg stat 2022-03-20T14:55:42.916 INFO:teuthology.orchestra.run.smithi007.stderr:pg getmap 2022-03-20T14:55:42.916 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-20T14:55:42.917 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-20T14:55:42.917 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_pools_json 2022-03-20T14:55:42.917 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-pool [...] 2022-03-20T14:55:42.918 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-primary [] [...] 2022-03-20T14:55:42.918 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-osd [] [...] 2022-03-20T14:55:42.918 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls [] [...] 2022-03-20T14:55:42.919 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-20T14:55:42.919 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2022-03-20T14:55:42.920 INFO:teuthology.orchestra.run.smithi007.stderr:command terminated with exit code 22 2022-03-20T14:55:42.922 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-20T14:55:42.960 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:42.961182 I | op-mon: mon c is not yet running 2022-03-20T14:55:42.960 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:42.961213 I | op-mon: mons running: [a b] 2022-03-20T14:55:47.978 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:47.979214 I | op-mon: mon c is not yet running 2022-03-20T14:55:47.978 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:47.979256 I | op-mon: mons running: [a b] 2022-03-20T14:55:52.924 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch status -f json 2022-03-20T14:55:53.739 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:53.740651 I | op-mon: mon c is not yet running 2022-03-20T14:55:53.739 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:53.740681 I | op-mon: mons running: [a b] 2022-03-20T14:55:54.092 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T14:55:54.093 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2022-03-20T14:55:54.093 INFO:teuthology.orchestra.run.smithi007.stderr:pg stat 2022-03-20T14:55:54.093 INFO:teuthology.orchestra.run.smithi007.stderr:pg getmap 2022-03-20T14:55:54.093 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-20T14:55:54.094 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-20T14:55:54.094 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_pools_json 2022-03-20T14:55:54.094 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-pool [...] 2022-03-20T14:55:54.094 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-primary [] [...] 2022-03-20T14:55:54.095 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-osd [] [...] 2022-03-20T14:55:54.095 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls [] [...] 2022-03-20T14:55:54.095 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-20T14:55:54.095 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2022-03-20T14:55:54.097 INFO:teuthology.orchestra.run.smithi007.stderr:command terminated with exit code 22 2022-03-20T14:55:54.100 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-20T14:55:58.823 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:58.825057 I | op-mon: mon c is not yet running 2022-03-20T14:55:58.823 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:55:58.825086 I | op-mon: mons running: [a b] 2022-03-20T14:56:03.844 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:03.846159 I | op-mon: mon c is not yet running 2022-03-20T14:56:03.845 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:03.846191 I | op-mon: mons running: [a b] 2022-03-20T14:56:04.101 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch status -f json 2022-03-20T14:56:04.819 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T14:56:04.820 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2022-03-20T14:56:04.820 INFO:teuthology.orchestra.run.smithi007.stderr:pg stat 2022-03-20T14:56:04.820 INFO:teuthology.orchestra.run.smithi007.stderr:pg getmap 2022-03-20T14:56:04.820 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-20T14:56:04.821 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-20T14:56:04.821 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_pools_json 2022-03-20T14:56:04.821 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-pool [...] 2022-03-20T14:56:04.821 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-primary [] [...] 2022-03-20T14:56:04.822 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-osd [] [...] 2022-03-20T14:56:04.822 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls [] [...] 2022-03-20T14:56:04.822 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-20T14:56:04.822 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2022-03-20T14:56:04.823 INFO:teuthology.orchestra.run.smithi007.stderr:command terminated with exit code 22 2022-03-20T14:56:04.825 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-20T14:56:08.868 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:08.866704 I | op-mon: mon c is not yet running 2022-03-20T14:56:08.869 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:08.866766 I | op-mon: mons running: [a b] 2022-03-20T14:56:14.457 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:14.459562 I | op-mon: mon c is not yet running 2022-03-20T14:56:14.458 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:14.459600 I | op-mon: mons running: [a b] 2022-03-20T14:56:14.827 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch status -f json 2022-03-20T14:56:16.283 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T14:56:16.284 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2022-03-20T14:56:16.284 INFO:teuthology.orchestra.run.smithi007.stderr:pg stat 2022-03-20T14:56:16.285 INFO:teuthology.orchestra.run.smithi007.stderr:pg getmap 2022-03-20T14:56:16.285 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-20T14:56:16.285 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-20T14:56:16.285 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_pools_json 2022-03-20T14:56:16.285 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-pool [...] 2022-03-20T14:56:16.286 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-primary [] [...] 2022-03-20T14:56:16.286 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-osd [] [...] 2022-03-20T14:56:16.286 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls [] [...] 2022-03-20T14:56:16.286 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-20T14:56:16.287 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2022-03-20T14:56:16.288 INFO:teuthology.orchestra.run.smithi007.stderr:command terminated with exit code 22 2022-03-20T14:56:16.290 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-20T14:56:19.654 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:19.655882 I | op-mon: mon c is not yet running 2022-03-20T14:56:19.654 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:19.655917 I | op-mon: mons running: [a b] 2022-03-20T14:56:25.219 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:25.221567 I | op-mon: mon c is not yet running 2022-03-20T14:56:25.220 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:25.221601 I | op-mon: mons running: [a b] 2022-03-20T14:56:26.291 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch status -f json 2022-03-20T14:56:26.852 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T14:56:26.853 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2022-03-20T14:56:26.853 INFO:teuthology.orchestra.run.smithi007.stderr:pg stat 2022-03-20T14:56:26.853 INFO:teuthology.orchestra.run.smithi007.stderr:pg getmap 2022-03-20T14:56:26.853 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-20T14:56:26.854 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-20T14:56:26.854 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_pools_json 2022-03-20T14:56:26.854 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-pool [...] 2022-03-20T14:56:26.854 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-primary [] [...] 2022-03-20T14:56:26.855 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-osd [] [...] 2022-03-20T14:56:26.855 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls [] [...] 2022-03-20T14:56:26.855 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-20T14:56:26.855 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2022-03-20T14:56:26.856 INFO:teuthology.orchestra.run.smithi007.stderr:command terminated with exit code 22 2022-03-20T14:56:26.857 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-20T14:56:30.268 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:30.270679 I | op-mon: mon c is not yet running 2022-03-20T14:56:30.269 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:30.270718 I | op-mon: mons running: [a b] 2022-03-20T14:56:35.312 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:35.313840 I | op-mon: mon c is not yet running 2022-03-20T14:56:35.312 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:35.313876 I | op-mon: mons running: [a b] 2022-03-20T14:56:36.859 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch status -f json 2022-03-20T14:56:40.333 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:40.336023 I | op-mon: mons running: [a b c] 2022-03-20T14:56:42.170 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T14:56:42.171 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2022-03-20T14:56:42.171 INFO:teuthology.orchestra.run.smithi007.stderr:pg stat 2022-03-20T14:56:42.171 INFO:teuthology.orchestra.run.smithi007.stderr:pg getmap 2022-03-20T14:56:42.172 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-20T14:56:42.172 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-20T14:56:42.172 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_pools_json 2022-03-20T14:56:42.172 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-pool [...] 2022-03-20T14:56:42.173 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-primary [] [...] 2022-03-20T14:56:42.173 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-osd [] [...] 2022-03-20T14:56:42.173 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls [] [...] 2022-03-20T14:56:42.173 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-20T14:56:42.173 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2022-03-20T14:56:42.174 INFO:teuthology.orchestra.run.smithi007.stderr:command terminated with exit code 22 2022-03-20T14:56:42.175 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-20T14:56:43.567 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:43.569243 I | op-mon: Monitors in quorum: [a b c] 2022-03-20T14:56:43.567 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:43.569281 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T14:56:43.784 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:43.786591 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T14:56:43.785 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:43.786629 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T14:56:44.009 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:44.011648 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T14:56:44.010 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:44.011680 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T14:56:44.283 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:44.285623 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T14:56:44.283 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:44.285655 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-20T14:56:44.537 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:44.539055 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-20T14:56:44.537 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:44.539095 I | op-config: deleting "log file" option from the mon configuration database 2022-03-20T14:56:44.756 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:44.758584 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-20T14:56:44.757 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:44.758631 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-20T14:56:45.079 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:45.081273 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-20T14:56:45.406 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:45.408572 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-20T14:56:45.815 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:45.817452 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-20T14:56:46.273 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:46.276041 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-20T14:56:46.274 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:46.276069 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-20T14:56:46.631 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:46.633751 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-20T14:56:46.632 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:46.633776 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-20T14:56:46.632 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:46.633780 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-20T14:56:47.017 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:47.019687 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-20T14:56:47.036 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:47.038657 I | op-mgr: start running mgr 2022-03-20T14:56:47.036 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:47.038703 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-20T14:56:47.703 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:56:47.706178 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi007": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T14:56:52.176 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch status -f json 2022-03-20T14:56:52.664 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T14:56:52.664 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2022-03-20T14:56:52.665 INFO:teuthology.orchestra.run.smithi007.stderr:pg stat 2022-03-20T14:56:52.665 INFO:teuthology.orchestra.run.smithi007.stderr:pg getmap 2022-03-20T14:56:52.665 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-20T14:56:52.665 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-20T14:56:52.666 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_pools_json 2022-03-20T14:56:52.666 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-pool [...] 2022-03-20T14:56:52.666 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-primary [] [...] 2022-03-20T14:56:52.667 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-osd [] [...] 2022-03-20T14:56:52.667 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls [] [...] 2022-03-20T14:56:52.667 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-20T14:56:52.668 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2022-03-20T14:56:52.668 INFO:teuthology.orchestra.run.smithi007.stderr:command terminated with exit code 22 2022-03-20T14:56:52.669 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-20T14:57:02.671 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch status -f json 2022-03-20T14:57:03.234 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T14:57:03.235 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENOENT: No orchestrator configured (try `ceph orch set backend`) 2022-03-20T14:57:03.241 INFO:teuthology.orchestra.run.smithi007.stderr:command terminated with exit code 2 2022-03-20T14:57:03.243 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-03-20T14:57:08.775 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:08.777890 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mgr-a" 2022-03-20T14:57:08.780 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:08.781815 I | op-mgr: setting services to point to mgr "a" 2022-03-20T14:57:08.837 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:08.839389 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-20T14:57:08.837 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:08.839502 I | op-mgr: successful modules: balancer 2022-03-20T14:57:08.902 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:08.903938 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-20T14:57:08.902 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:08.903961 W | op-osd: useAllNodes is set to false and no nodes, storageClassDevicesets or volumeSources are specified, no OSD pods are going to be created 2022-03-20T14:57:08.902 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:08.903972 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-20T14:57:08.909 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:08.912895 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-20T14:57:08.918 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:08.921146 I | op-osd: no storageClassDeviceSets defined to configure OSDs on PVCs 2022-03-20T14:57:08.918 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:08.921173 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-20T14:57:08.918 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:08.921179 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-20T14:57:09.371 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:09.373544 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-20T14:57:09.371 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:09.373577 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-20T14:57:09.380 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:09.382790 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph" 2022-03-20T14:57:09.380 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:09.382819 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "rook-ceph" 2022-03-20T14:57:09.380 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:09.382845 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph" 2022-03-20T14:57:09.425 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:09.428246 I | op-mgr: successful modules: prometheus 2022-03-20T14:57:09.427 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:09.428906 I | op-mgr: successful modules: dashboard 2022-03-20T14:57:09.464 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:09.467263 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-20T14:57:13.245 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch status -f json 2022-03-20T14:57:16.221 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:16.224707 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi057": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T14:57:16.327 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:16.329946 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi057": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T14:57:19.048 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T14:57:19.051 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENOENT: No orchestrator configured (try `ceph orch set backend`) 2022-03-20T14:57:19.057 INFO:teuthology.orchestra.run.smithi007.stderr:command terminated with exit code 2 2022-03-20T14:57:19.058 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-03-20T14:57:19.478 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:19.481525 I | op-mgr: successful modules: orchestrator modules 2022-03-20T14:57:29.060 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch status -f json 2022-03-20T14:57:29.723 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T14:57:29.724 INFO:teuthology.orchestra.run.smithi007.stdout:{"available": true, "backend": "rook"} 2022-03-20T14:57:29.747 INFO:tasks.rook: mgr/rook orchestrator is active 2022-03-20T14:57:29.747 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph config set mgr mgr/rook/storage_class scratch 2022-03-20T14:57:30.410 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch apply osd --all-available-devices 2022-03-20T14:57:31.214 INFO:teuthology.orchestra.run.smithi007.stdout:['Success'] 2022-03-20T14:57:31.224 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.219306 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{ 2022-03-20T14:57:31.224 INFO:tasks.rook.operator.smithi007.stdout:   CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", AllowUnsupported: true}, 2022-03-20T14:57:31.224 INFO:tasks.rook.operator.smithi007.stdout:   Storage: v1.StorageScopeSpec{ 2022-03-20T14:57:31.225 INFO:tasks.rook.operator.smithi007.stdout:   ... // 3 identical fields 2022-03-20T14:57:31.225 INFO:tasks.rook.operator.smithi007.stdout:   Config: nil, 2022-03-20T14:57:31.225 INFO:tasks.rook.operator.smithi007.stdout:   Selection: {}, 2022-03-20T14:57:31.225 INFO:tasks.rook.operator.smithi007.stdout:-  StorageClassDeviceSets: nil, 2022-03-20T14:57:31.226 INFO:tasks.rook.operator.smithi007.stdout:+  StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-20T14:57:31.226 INFO:tasks.rook.operator.smithi007.stdout:+  { 2022-03-20T14:57:31.226 INFO:tasks.rook.operator.smithi007.stdout:+  Name: "smithi007-lv-1", 2022-03-20T14:57:31.226 INFO:tasks.rook.operator.smithi007.stdout:+  Count: 1, 2022-03-20T14:57:31.227 INFO:tasks.rook.operator.smithi007.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-20T14:57:31.227 INFO:tasks.rook.operator.smithi007.stdout:+  }, 2022-03-20T14:57:31.227 INFO:tasks.rook.operator.smithi007.stdout:+  { 2022-03-20T14:57:31.227 INFO:tasks.rook.operator.smithi007.stdout:+  Name: "smithi007-lv-2", 2022-03-20T14:57:31.228 INFO:tasks.rook.operator.smithi007.stdout:+  Count: 1, 2022-03-20T14:57:31.228 INFO:tasks.rook.operator.smithi007.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-20T14:57:31.228 INFO:tasks.rook.operator.smithi007.stdout:+  }, 2022-03-20T14:57:31.228 INFO:tasks.rook.operator.smithi007.stdout:+  { 2022-03-20T14:57:31.229 INFO:tasks.rook.operator.smithi007.stdout:+  Name: "smithi007-lv-3", 2022-03-20T14:57:31.229 INFO:tasks.rook.operator.smithi007.stdout:+  Count: 1, 2022-03-20T14:57:31.229 INFO:tasks.rook.operator.smithi007.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-20T14:57:31.229 INFO:tasks.rook.operator.smithi007.stdout:+  }, 2022-03-20T14:57:31.229 INFO:tasks.rook.operator.smithi007.stdout:+  { 2022-03-20T14:57:31.230 INFO:tasks.rook.operator.smithi007.stdout:+  Name: "smithi007-lv-4", 2022-03-20T14:57:31.230 INFO:tasks.rook.operator.smithi007.stdout:+  Count: 1, 2022-03-20T14:57:31.230 INFO:tasks.rook.operator.smithi007.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-20T14:57:31.230 INFO:tasks.rook.operator.smithi007.stdout:+  }, 2022-03-20T14:57:31.231 INFO:tasks.rook.operator.smithi007.stdout:+  { 2022-03-20T14:57:31.231 INFO:tasks.rook.operator.smithi007.stdout:+  Name: "smithi057-lv-1", 2022-03-20T14:57:31.231 INFO:tasks.rook.operator.smithi007.stdout:+  Count: 1, 2022-03-20T14:57:31.231 INFO:tasks.rook.operator.smithi007.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-20T14:57:31.231 INFO:tasks.rook.operator.smithi007.stdout:+  }, 2022-03-20T14:57:31.232 INFO:tasks.rook.operator.smithi007.stdout:+  { 2022-03-20T14:57:31.232 INFO:tasks.rook.operator.smithi007.stdout:+  Name: "smithi057-lv-2", 2022-03-20T14:57:31.232 INFO:tasks.rook.operator.smithi007.stdout:+  Count: 1, 2022-03-20T14:57:31.232 INFO:tasks.rook.operator.smithi007.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-20T14:57:31.233 INFO:tasks.rook.operator.smithi007.stdout:+  }, 2022-03-20T14:57:31.233 INFO:tasks.rook.operator.smithi007.stdout:+  { 2022-03-20T14:57:31.233 INFO:tasks.rook.operator.smithi007.stdout:+  Name: "smithi057-lv-3", 2022-03-20T14:57:31.233 INFO:tasks.rook.operator.smithi007.stdout:+  Count: 1, 2022-03-20T14:57:31.233 INFO:tasks.rook.operator.smithi007.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-20T14:57:31.234 INFO:tasks.rook.operator.smithi007.stdout:+  }, 2022-03-20T14:57:31.234 INFO:tasks.rook.operator.smithi007.stdout:+  { 2022-03-20T14:57:31.234 INFO:tasks.rook.operator.smithi007.stdout:+  Name: "smithi057-lv-4", 2022-03-20T14:57:31.234 INFO:tasks.rook.operator.smithi007.stdout:+  Count: 1, 2022-03-20T14:57:31.235 INFO:tasks.rook.operator.smithi007.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-20T14:57:31.235 INFO:tasks.rook.operator.smithi007.stdout:+  }, 2022-03-20T14:57:31.235 INFO:tasks.rook.operator.smithi007.stdout:+  { 2022-03-20T14:57:31.235 INFO:tasks.rook.operator.smithi007.stdout:+  Name: "smithi087-lv-1", 2022-03-20T14:57:31.236 INFO:tasks.rook.operator.smithi007.stdout:+  Count: 1, 2022-03-20T14:57:31.236 INFO:tasks.rook.operator.smithi007.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-20T14:57:31.236 INFO:tasks.rook.operator.smithi007.stdout:+  }, 2022-03-20T14:57:31.236 INFO:tasks.rook.operator.smithi007.stdout:+  { 2022-03-20T14:57:31.236 INFO:tasks.rook.operator.smithi007.stdout:+  Name: "smithi087-lv-2", 2022-03-20T14:57:31.237 INFO:tasks.rook.operator.smithi007.stdout:+  Count: 1, 2022-03-20T14:57:31.237 INFO:tasks.rook.operator.smithi007.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-20T14:57:31.237 INFO:tasks.rook.operator.smithi007.stdout:+  }, 2022-03-20T14:57:31.237 INFO:tasks.rook.operator.smithi007.stdout:+  { 2022-03-20T14:57:31.238 INFO:tasks.rook.operator.smithi007.stdout:+  Name: "smithi087-lv-3", 2022-03-20T14:57:31.238 INFO:tasks.rook.operator.smithi007.stdout:+  Count: 1, 2022-03-20T14:57:31.238 INFO:tasks.rook.operator.smithi007.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-20T14:57:31.238 INFO:tasks.rook.operator.smithi007.stdout:+  }, 2022-03-20T14:57:31.238 INFO:tasks.rook.operator.smithi007.stdout:+  { 2022-03-20T14:57:31.239 INFO:tasks.rook.operator.smithi007.stdout:+  Name: "smithi087-lv-4", 2022-03-20T14:57:31.239 INFO:tasks.rook.operator.smithi007.stdout:+  Count: 1, 2022-03-20T14:57:31.239 INFO:tasks.rook.operator.smithi007.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-20T14:57:31.239 INFO:tasks.rook.operator.smithi007.stdout:+  }, 2022-03-20T14:57:31.240 INFO:tasks.rook.operator.smithi007.stdout:+  }, 2022-03-20T14:57:31.240 INFO:tasks.rook.operator.smithi007.stdout:   }, 2022-03-20T14:57:31.240 INFO:tasks.rook.operator.smithi007.stdout:   Annotations: nil, 2022-03-20T14:57:31.240 INFO:tasks.rook.operator.smithi007.stdout:   Labels: nil, 2022-03-20T14:57:31.240 INFO:tasks.rook.operator.smithi007.stdout:   ... // 20 identical fields 2022-03-20T14:57:31.241 INFO:tasks.rook.operator.smithi007.stdout:  } 2022-03-20T14:57:31.241 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.219466 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-20T14:57:31.241 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.219922 I | operator: successfully started the controller-runtime manager 2022-03-20T14:57:31.241 INFO:tasks.rook.operator.smithi007.stdout:I0320 14:57:31.219984 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-20T14:57:31.242 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.219986 I | op-mon: stopping monitoring of mons in namespace "rook-ceph" 2022-03-20T14:57:31.242 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.220015 I | ceph-cluster-controller: stopping monitoring of ceph status 2022-03-20T14:57:31.242 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.220035 I | op-osd: stopping monitoring of OSDs in namespace "rook-ceph" 2022-03-20T14:57:31.245 INFO:tasks.rook:Waiting for 12 OSDs 2022-03-20T14:57:31.245 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd stat -f json 2022-03-20T14:57:31.254 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.257278 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-20T14:57:31.254 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.257306 I | operator: watching all namespaces for Ceph CRs 2022-03-20T14:57:31.254 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.257361 I | operator: setting up schemes 2022-03-20T14:57:31.258 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.260731 I | operator: setting up the controller-runtime manager 2022-03-20T14:57:31.461 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.463891 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-20T14:57:31.465 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.466475 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-20T14:57:31.465 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.466518 I | ceph-cluster-controller: successfully started 2022-03-20T14:57:31.466 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.466600 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-20T14:57:31.466 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.466643 I | ceph-crashcollector-controller: successfully started 2022-03-20T14:57:31.466 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.466689 I | ceph-block-pool-controller: successfully started 2022-03-20T14:57:31.466 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.466729 I | ceph-object-store-user-controller: successfully started 2022-03-20T14:57:31.467 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.466765 I | ceph-object-realm-controller: successfully started 2022-03-20T14:57:31.467 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.466784 I | ceph-object-zonegroup-controller: successfully started 2022-03-20T14:57:31.467 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.466798 I | ceph-object-zone-controller: successfully started 2022-03-20T14:57:31.467 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.466960 I | ceph-object-controller: successfully started 2022-03-20T14:57:31.468 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.467013 I | ceph-file-controller: successfully started 2022-03-20T14:57:31.468 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.467076 I | ceph-nfs-controller: successfully started 2022-03-20T14:57:31.469 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.467151 I | ceph-rbd-mirror-controller: successfully started 2022-03-20T14:57:31.469 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.467212 I | ceph-client-controller: successfully started 2022-03-20T14:57:31.469 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.467257 I | ceph-filesystem-mirror-controller: successfully started 2022-03-20T14:57:31.469 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.467322 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-20T14:57:31.470 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.467362 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-20T14:57:31.470 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.467401 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-20T14:57:31.470 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.467439 I | ceph-bucket-topic: successfully started 2022-03-20T14:57:31.470 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.467473 I | ceph-bucket-notification: successfully started 2022-03-20T14:57:31.471 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.467496 I | ceph-bucket-notification: successfully started 2022-03-20T14:57:31.471 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.467525 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-20T14:57:31.471 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.468842 I | operator: starting the controller-runtime manager 2022-03-20T14:57:31.570 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.572909 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-20T14:57:31.570 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.572972 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-20T14:57:31.571 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.572989 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-20T14:57:31.574 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.573998 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-20T14:57:31.578 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.577485 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-20T14:57:31.578 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.580183 I | op-mon: parsing mon endpoints: a=10.98.115.144:6789,b=10.109.19.50:6789,c=10.106.65.43:6789 2022-03-20T14:57:31.578 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.580531 I | op-mon: parsing mon endpoints: a=10.98.115.144:6789,b=10.109.19.50:6789,c=10.106.65.43:6789 2022-03-20T14:57:31.578 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.580578 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-20T14:57:31.579 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.580589 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-20T14:57:31.580 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.582168 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-20T14:57:31.581 INFO:tasks.rook.operator.smithi007.stdout:I0320 14:57:31.582254 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-20T14:57:31.609 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.612579 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-20T14:57:31.614 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.617010 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e... 2022-03-20T14:57:31.658 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.659886 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-20T14:57:31.658 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.659929 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-20T14:57:31.658 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.659945 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-20T14:57:31.659 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.659957 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-20T14:57:31.659 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.659969 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-20T14:57:31.659 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.659983 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-20T14:57:31.659 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.659998 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-20T14:57:31.660 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.660012 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-20T14:57:31.660 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.660027 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-20T14:57:31.660 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.660041 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-20T14:57:31.660 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.660054 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-20T14:57:31.660 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.660069 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-20T14:57:31.661 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.660086 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-20T14:57:31.661 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.660097 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-20T14:57:31.661 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.660118 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-20T14:57:31.661 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.660136 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-20T14:57:31.662 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.660228 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-20T14:57:31.662 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:31.660256 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-20T14:57:31.771 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T14:57:31.793 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-20T14:57:31.793 INFO:tasks.rook: have 0/12 OSDs 2022-03-20T14:57:38.837 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:38.839517 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-20T14:57:38.837 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:38.839560 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-20T14:57:38.837 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:38.839577 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-20T14:57:38.838 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:38.839590 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-20T14:57:38.838 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:38.839603 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-20T14:57:38.838 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:38.839616 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-20T14:57:38.838 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:38.839631 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-20T14:57:38.839 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:38.839644 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-20T14:57:38.841 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:38.839658 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-20T14:57:38.841 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:38.839672 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-20T14:57:38.841 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:38.839684 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-20T14:57:38.841 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:38.839698 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-20T14:57:38.842 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:38.839711 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-20T14:57:38.842 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:38.839725 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-20T14:57:38.842 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:38.839742 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-20T14:57:38.842 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:38.839755 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-20T14:57:38.842 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:38.839768 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-20T14:57:38.843 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:38.839780 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-20T14:57:38.843 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:38.839793 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-20T14:57:38.843 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:38.839878 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-20T14:57:38.843 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:38.839908 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-20T14:57:38.844 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:38.839917 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-20T14:57:38.844 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:38.839924 I | ceph-csi: Kubernetes version is 1.23 2022-03-20T14:57:38.844 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:38.839939 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-20T14:57:38.844 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:38.839956 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-20T14:57:38.845 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:38.844768 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-20T14:57:38.848 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:38.851306 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-20T14:57:38.849 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:38.851328 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-20T14:57:38.849 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:38.851334 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-20T14:57:38.849 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:38.851339 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-20T14:57:38.849 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:38.851343 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-20T14:57:38.850 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:38.851347 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-20T14:57:38.850 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:38.851351 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-20T14:57:38.987 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:38.990583 I | ceph-spec: detected ceph image version: "17.0.0-11187 quincy" 2022-03-20T14:57:38.988 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:38.990603 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-20T14:57:39.009 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:39.011705 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-20T14:57:39.009 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:39.011735 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-20T14:57:39.009 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:39.011744 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-20T14:57:39.033 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:39.036870 I | op-mon: parsing mon endpoints: a=10.98.115.144:6789,b=10.109.19.50:6789,c=10.106.65.43:6789 2022-03-20T14:57:39.145 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:39.147567 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T14:57:39.151 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:39.150327 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T14:57:39.178 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:39.181310 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-20T14:57:39.407 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:39.410308 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-20T14:57:39.407 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:39.410336 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-20T14:57:39.408 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:39.410345 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-20T14:57:39.514 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:39.517720 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-20T14:57:39.514 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:39.517736 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-20T14:57:39.515 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:39.517740 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-20T14:57:39.565 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:39.568875 I | ceph-csi: successfully started CSI CephFS driver 2022-03-20T14:57:39.656 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:39.659769 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11187 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e" 2022-03-20T14:57:39.846 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:39.849022 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-20T14:57:39.860 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:39.863634 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-20T14:57:39.866 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:39.863665 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-20T14:57:39.890 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:39.893942 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-20T14:57:39.930 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:39.933554 I | op-mon: start running mons 2022-03-20T14:57:40.164 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:40.167492 I | op-mon: parsing mon endpoints: a=10.98.115.144:6789,b=10.109.19.50:6789,c=10.106.65.43:6789 2022-03-20T14:57:41.169 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:41.172082 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.98.115.144:6789","10.109.19.50:6789","10.106.65.43:6789"]}] data:a=10.98.115.144:6789,b=10.109.19.50:6789,c=10.106.65.43:6789 mapping:{"node":{"a":{"Name":"smithi087","Hostname":"smithi087","Address":"172.21.15.87"},"b":{"Name":"smithi057","Hostname":"smithi057","Address":"172.21.15.57"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:2] 2022-03-20T14:57:41.764 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:41.767775 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T14:57:41.765 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:41.768068 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T14:57:41.794 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd stat -f json 2022-03-20T14:57:42.312 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T14:57:42.340 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-20T14:57:42.340 INFO:tasks.rook: have 0/12 OSDs 2022-03-20T14:57:43.365 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:43.368161 I | op-mon: targeting the mon count 3 2022-03-20T14:57:43.370 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:43.373567 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T14:57:43.739 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:43.742381 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T14:57:43.740 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:43.742417 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T14:57:44.091 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:44.092609 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T14:57:44.092 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:44.092649 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T14:57:44.464 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:44.466988 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T14:57:44.464 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:44.467023 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-20T14:57:44.822 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:44.825893 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-20T14:57:44.823 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:44.825927 I | op-config: deleting "log file" option from the mon configuration database 2022-03-20T14:57:45.171 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:45.174288 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-20T14:57:45.171 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:45.174321 I | op-mon: checking for basic quorum with existing mons 2022-03-20T14:57:45.333 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:45.336195 I | op-mon: mon "a" ip is 10.98.115.144 2022-03-20T14:57:45.496 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:45.499439 I | op-mon: mon "b" ip is 10.109.19.50 2022-03-20T14:57:45.777 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:45.780003 I | op-mon: mon "c" ip is 10.106.65.43 2022-03-20T14:57:46.365 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:46.369113 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.98.115.144:6789","10.109.19.50:6789","10.106.65.43:6789"]}] data:a=10.98.115.144:6789,b=10.109.19.50:6789,c=10.106.65.43:6789 mapping:{"node":{"a":{"Name":"smithi087","Hostname":"smithi087","Address":"172.21.15.87"},"b":{"Name":"smithi057","Hostname":"smithi057","Address":"172.21.15.57"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:2] 2022-03-20T14:57:46.964 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:46.967347 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T14:57:46.965 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:46.967560 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T14:57:47.387 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:47.390242 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-20T14:57:47.395 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:47.399200 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-20T14:57:47.396 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:47.399230 I | op-mon: waiting for mon quorum with [a b c] 2022-03-20T14:57:47.970 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:47.973545 I | op-mon: mons running: [a b c] 2022-03-20T14:57:48.397 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:48.400870 I | op-mon: Monitors in quorum: [a b c] 2022-03-20T14:57:48.411 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:48.414783 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-20T14:57:48.422 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:48.426582 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-20T14:57:48.423 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:48.426606 I | op-mon: waiting for mon quorum with [a b c] 2022-03-20T14:57:48.569 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:48.572911 I | op-mon: mons running: [a b c] 2022-03-20T14:57:49.007 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:49.010340 I | op-mon: Monitors in quorum: [a b c] 2022-03-20T14:57:49.014 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:49.017635 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-20T14:57:49.024 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:49.027750 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-20T14:57:49.024 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:49.027775 I | op-mon: waiting for mon quorum with [a b c] 2022-03-20T14:57:49.169 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:49.173034 I | op-mon: mons running: [a b c] 2022-03-20T14:57:49.578 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:49.581692 I | op-mon: Monitors in quorum: [a b c] 2022-03-20T14:57:49.578 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:49.581722 I | op-mon: mons created: 3 2022-03-20T14:57:49.992 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:49.995467 I | op-mon: waiting for mon quorum with [a b c] 2022-03-20T14:57:50.016 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:50.020057 I | op-mon: mons running: [a b c] 2022-03-20T14:57:50.464 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:50.467178 I | op-mon: Monitors in quorum: [a b c] 2022-03-20T14:57:50.464 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:50.467224 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-20T14:57:50.886 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:50.889417 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-20T14:57:51.298 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:51.301329 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-20T14:57:51.714 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:51.717109 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-20T14:57:52.161 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:52.165097 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-20T14:57:52.162 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:52.165123 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-20T14:57:52.341 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd stat -f json 2022-03-20T14:57:52.596 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:52.599788 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-20T14:57:52.597 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:52.599844 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-20T14:57:52.597 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:52.599854 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-20T14:57:52.966 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T14:57:52.986 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-20T14:57:52.986 INFO:tasks.rook: have 0/12 OSDs 2022-03-20T14:57:53.145 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:53.148876 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-20T14:57:53.169 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:53.173175 I | op-mgr: start running mgr 2022-03-20T14:57:53.170 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:53.173210 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-20T14:57:53.642 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:53.645371 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-20T14:57:53.651 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:53.654787 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-20T14:57:53.655 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:53.658966 I | op-mgr: setting services to point to mgr "a" 2022-03-20T14:57:54.137 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:54.140401 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-20T14:57:54.137 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:54.140590 I | op-mgr: successful modules: balancer 2022-03-20T14:57:54.152 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:54.155933 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-20T14:57:54.152 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:54.155971 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-20T14:57:54.168 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:54.172220 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-20T14:57:54.337 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:54.340980 I | op-osd: creating 1 new PVCs for device set "smithi007-lv-1" 2022-03-20T14:57:54.540 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:54.544067 I | op-osd: successfully provisioned PVC "smithi007-lv-1-data-0fltgg" 2022-03-20T14:57:54.541 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:54.544097 I | op-osd: creating 1 new PVCs for device set "smithi007-lv-2" 2022-03-20T14:57:54.756 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:54.759498 I | op-osd: successfully provisioned PVC "smithi007-lv-2-data-0z6lbn" 2022-03-20T14:57:54.756 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:54.759537 I | op-osd: creating 1 new PVCs for device set "smithi007-lv-3" 2022-03-20T14:57:54.944 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:54.947335 I | op-osd: successfully provisioned PVC "smithi007-lv-3-data-0wb6kn" 2022-03-20T14:57:54.944 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:54.947375 I | op-osd: creating 1 new PVCs for device set "smithi007-lv-4" 2022-03-20T14:57:55.158 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:55.161552 I | op-osd: successfully provisioned PVC "smithi007-lv-4-data-0snpfz" 2022-03-20T14:57:55.158 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:55.161593 I | op-osd: creating 1 new PVCs for device set "smithi057-lv-1" 2022-03-20T14:57:55.362 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:55.365493 I | op-osd: successfully provisioned PVC "smithi057-lv-1-data-056qwk" 2022-03-20T14:57:55.362 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:55.365527 I | op-osd: creating 1 new PVCs for device set "smithi057-lv-2" 2022-03-20T14:57:55.576 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:55.579911 I | op-osd: successfully provisioned PVC "smithi057-lv-2-data-0xdlsb" 2022-03-20T14:57:55.576 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:55.579963 I | op-osd: creating 1 new PVCs for device set "smithi057-lv-3" 2022-03-20T14:57:55.676 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:55.679314 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-20T14:57:55.679 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:55.681854 I | op-mgr: successful modules: dashboard 2022-03-20T14:57:55.681 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:55.684010 I | op-mgr: successful modules: prometheus 2022-03-20T14:57:55.737 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:55.741029 I | op-osd: successfully provisioned PVC "smithi057-lv-3-data-0vrztn" 2022-03-20T14:57:55.738 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:55.741062 I | op-osd: creating 1 new PVCs for device set "smithi057-lv-4" 2022-03-20T14:57:55.941 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:55.945074 I | op-osd: successfully provisioned PVC "smithi057-lv-4-data-0zt8b4" 2022-03-20T14:57:55.942 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:55.945107 I | op-osd: creating 1 new PVCs for device set "smithi087-lv-1" 2022-03-20T14:57:56.137 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:56.140868 I | op-osd: successfully provisioned PVC "smithi087-lv-1-data-0p7bb2" 2022-03-20T14:57:56.138 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:56.140928 I | op-osd: creating 1 new PVCs for device set "smithi087-lv-2" 2022-03-20T14:57:56.338 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:56.342144 I | op-osd: successfully provisioned PVC "smithi087-lv-2-data-04lndq" 2022-03-20T14:57:56.339 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:56.342179 I | op-osd: creating 1 new PVCs for device set "smithi087-lv-3" 2022-03-20T14:57:56.541 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:56.545035 I | op-osd: successfully provisioned PVC "smithi087-lv-3-data-0brh6f" 2022-03-20T14:57:56.542 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:56.545090 I | op-osd: creating 1 new PVCs for device set "smithi087-lv-4" 2022-03-20T14:57:56.738 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:56.741786 I | op-osd: successfully provisioned PVC "smithi087-lv-4-data-0vssdn" 2022-03-20T14:57:56.749 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:56.753339 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-1-data-0fltgg" 2022-03-20T14:57:57.153 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:57.156651 I | op-osd: started OSD provisioning job for PVC "smithi007-lv-1-data-0fltgg" 2022-03-20T14:57:57.153 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:57.156689 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-2-data-0z6lbn" 2022-03-20T14:57:57.439 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:57.442492 I | op-mgr: successful modules: orchestrator modules 2022-03-20T14:57:57.553 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:57.556771 I | op-osd: started OSD provisioning job for PVC "smithi007-lv-2-data-0z6lbn" 2022-03-20T14:57:57.553 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:57.556802 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-3-data-0wb6kn" 2022-03-20T14:57:58.249 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:58.252723 I | op-osd: started OSD provisioning job for PVC "smithi007-lv-3-data-0wb6kn" 2022-03-20T14:57:58.250 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:58.252760 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-4-data-0snpfz" 2022-03-20T14:57:58.627 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:58.630148 I | op-osd: started OSD provisioning job for PVC "smithi007-lv-4-data-0snpfz" 2022-03-20T14:57:58.627 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:58.630185 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-1-data-056qwk" 2022-03-20T14:57:59.255 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:59.258228 I | op-osd: started OSD provisioning job for PVC "smithi057-lv-1-data-056qwk" 2022-03-20T14:57:59.256 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:59.258265 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-2-data-0xdlsb" 2022-03-20T14:57:59.560 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:59.563523 I | op-osd: started OSD provisioning job for PVC "smithi057-lv-2-data-0xdlsb" 2022-03-20T14:57:59.560 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:57:59.563560 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-3-data-0vrztn" 2022-03-20T14:58:00.384 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:00.387513 I | op-osd: started OSD provisioning job for PVC "smithi057-lv-3-data-0vrztn" 2022-03-20T14:58:00.384 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:00.387553 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-4-data-0zt8b4" 2022-03-20T14:58:01.484 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:01.487787 I | op-osd: started OSD provisioning job for PVC "smithi057-lv-4-data-0zt8b4" 2022-03-20T14:58:01.484 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:01.487855 I | op-osd: OSD will have its main bluestore block on "smithi087-lv-1-data-0p7bb2" 2022-03-20T14:58:02.548 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:02.551560 I | op-osd: started OSD provisioning job for PVC "smithi087-lv-1-data-0p7bb2" 2022-03-20T14:58:02.548 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:02.551590 I | op-osd: OSD will have its main bluestore block on "smithi087-lv-2-data-04lndq" 2022-03-20T14:58:02.988 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd stat -f json 2022-03-20T14:58:03.529 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:03.533030 I | op-osd: started OSD provisioning job for PVC "smithi087-lv-2-data-04lndq" 2022-03-20T14:58:03.530 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:03.533067 I | op-osd: OSD will have its main bluestore block on "smithi087-lv-3-data-0brh6f" 2022-03-20T14:58:04.192 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T14:58:04.209 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-20T14:58:04.209 INFO:tasks.rook: have 0/12 OSDs 2022-03-20T14:58:04.376 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:04.379755 I | op-osd: started OSD provisioning job for PVC "smithi087-lv-3-data-0brh6f" 2022-03-20T14:58:04.376 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:04.379795 I | op-osd: OSD will have its main bluestore block on "smithi087-lv-4-data-0vssdn" 2022-03-20T14:58:04.943 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:04.947074 I | op-osd: started OSD provisioning job for PVC "smithi087-lv-4-data-0vssdn" 2022-03-20T14:58:04.943 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:04.947105 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-20T14:58:04.944 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:04.947111 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-20T14:58:06.283 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:06.286437 I | op-osd: OSD orchestration status for PVC smithi007-lv-1-data-0fltgg is "starting" 2022-03-20T14:58:06.284 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:06.286459 I | op-osd: OSD orchestration status for PVC smithi007-lv-2-data-0z6lbn is "starting" 2022-03-20T14:58:06.284 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:06.286468 I | op-osd: OSD orchestration status for PVC smithi007-lv-3-data-0wb6kn is "starting" 2022-03-20T14:58:06.284 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:06.286475 I | op-osd: OSD orchestration status for PVC smithi007-lv-4-data-0snpfz is "starting" 2022-03-20T14:58:06.285 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:06.286482 I | op-osd: OSD orchestration status for PVC smithi057-lv-1-data-056qwk is "starting" 2022-03-20T14:58:06.285 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:06.286492 I | op-osd: OSD orchestration status for PVC smithi057-lv-2-data-0xdlsb is "starting" 2022-03-20T14:58:06.285 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:06.286505 I | op-osd: OSD orchestration status for PVC smithi057-lv-3-data-0vrztn is "starting" 2022-03-20T14:58:06.285 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:06.286517 I | op-osd: OSD orchestration status for PVC smithi057-lv-4-data-0zt8b4 is "starting" 2022-03-20T14:58:06.285 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:06.286534 I | op-osd: OSD orchestration status for PVC smithi087-lv-1-data-0p7bb2 is "starting" 2022-03-20T14:58:06.286 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:06.286545 I | op-osd: OSD orchestration status for PVC smithi087-lv-2-data-04lndq is "starting" 2022-03-20T14:58:06.286 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:06.286556 I | op-osd: OSD orchestration status for PVC smithi087-lv-3-data-0brh6f is "starting" 2022-03-20T14:58:06.286 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:06.286568 I | op-osd: OSD orchestration status for PVC smithi087-lv-4-data-0vssdn is "starting" 2022-03-20T14:58:14.210 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd stat -f json 2022-03-20T14:58:15.922 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T14:58:15.944 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-20T14:58:15.945 INFO:tasks.rook: have 0/12 OSDs 2022-03-20T14:58:16.545 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:16.548462 I | op-osd: OSD orchestration status for node smithi057-lv-2-data-0xdlsb is "orchestrating" 2022-03-20T14:58:16.550 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:16.549241 I | op-osd: OSD orchestration status for node smithi057-lv-3-data-0vrztn is "orchestrating" 2022-03-20T14:58:16.704 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:16.708317 I | op-osd: OSD orchestration status for node smithi057-lv-1-data-056qwk is "orchestrating" 2022-03-20T14:58:16.722 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:16.725765 I | op-osd: OSD orchestration status for node smithi057-lv-4-data-0zt8b4 is "orchestrating" 2022-03-20T14:58:17.079 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:17.083219 I | op-osd: OSD orchestration status for PVC smithi057-lv-2-data-0xdlsb is "orchestrating" 2022-03-20T14:58:17.146 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:17.150010 I | op-osd: OSD orchestration status for PVC smithi057-lv-3-data-0vrztn is "orchestrating" 2022-03-20T14:58:17.166 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:17.170078 I | op-osd: OSD orchestration status for PVC smithi057-lv-1-data-056qwk is "orchestrating" 2022-03-20T14:58:17.174 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:17.178302 I | op-osd: OSD orchestration status for PVC smithi057-lv-4-data-0zt8b4 is "orchestrating" 2022-03-20T14:58:18.045 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:18.049491 I | op-osd: OSD orchestration status for node smithi087-lv-1-data-0p7bb2 is "orchestrating" 2022-03-20T14:58:19.170 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:19.174104 I | op-osd: OSD orchestration status for PVC smithi087-lv-1-data-0p7bb2 is "orchestrating" 2022-03-20T14:58:19.966 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:19.969711 I | op-osd: OSD orchestration status for node smithi007-lv-4-data-0snpfz is "orchestrating" 2022-03-20T14:58:20.379 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:20.383518 I | op-osd: OSD orchestration status for node smithi087-lv-3-data-0brh6f is "orchestrating" 2022-03-20T14:58:20.380 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:20.383690 I | op-osd: OSD orchestration status for node smithi087-lv-2-data-04lndq is "orchestrating" 2022-03-20T14:58:20.384 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:20.383870 I | op-osd: OSD orchestration status for node smithi087-lv-4-data-0vssdn is "orchestrating" 2022-03-20T14:58:20.725 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:20.728406 I | op-osd: OSD orchestration status for PVC smithi007-lv-4-data-0snpfz is "orchestrating" 2022-03-20T14:58:20.743 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:20.747107 I | op-osd: OSD orchestration status for node smithi007-lv-1-data-0fltgg is "orchestrating" 2022-03-20T14:58:20.749 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:20.753887 I | op-osd: OSD orchestration status for node smithi007-lv-3-data-0wb6kn is "orchestrating" 2022-03-20T14:58:21.017 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:21.021328 I | op-osd: OSD orchestration status for PVC smithi087-lv-3-data-0brh6f is "orchestrating" 2022-03-20T14:58:21.101 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:21.104891 I | op-osd: OSD orchestration status for PVC smithi087-lv-2-data-04lndq is "orchestrating" 2022-03-20T14:58:21.101 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:21.104942 I | op-osd: OSD orchestration status for PVC smithi087-lv-4-data-0vssdn is "orchestrating" 2022-03-20T14:58:21.294 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:21.298783 I | op-osd: OSD orchestration status for PVC smithi007-lv-1-data-0fltgg is "orchestrating" 2022-03-20T14:58:21.321 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:21.324910 I | op-osd: OSD orchestration status for PVC smithi007-lv-3-data-0wb6kn is "orchestrating" 2022-03-20T14:58:21.326 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:21.330881 I | op-osd: OSD orchestration status for node smithi007-lv-2-data-0z6lbn is "orchestrating" 2022-03-20T14:58:21.740 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:21.744341 I | op-osd: OSD orchestration status for PVC smithi007-lv-2-data-0z6lbn is "orchestrating" 2022-03-20T14:58:22.341 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:22.335573 I | op-osd: OSD orchestration status for PVC smithi087-lv-3-data-0brh6f is "failed" 2022-03-20T14:58:22.341 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:22.335626 E | op-osd: failed to provision OSD(s) on PVC smithi087-lv-3-data-0brh6f. &{OSDs:[] Status:failed PvcBackedOSD:true Message:failed to configure devices: failed to initialize devices on PVC: failed to run ceph-volume. Running command: /usr/bin/ceph-authtool --gen-print-key 2022-03-20T14:58:22.341 INFO:tasks.rook.operator.smithi007.stdout:Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e1856f16-99e2-4716-b7b5-bcf6c397f73d 2022-03-20T14:58:22.342 INFO:tasks.rook.operator.smithi007.stdout: stderr: unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-20T14:58:22.342 INFO:tasks.rook.operator.smithi007.stdout: stderr: 2022-03-20T14:58:22.286+0000 7fd500727700 -1 failed for service _ceph-mon._tcp 2022-03-20T14:58:22.342 INFO:tasks.rook.operator.smithi007.stdout: stderr: 2022-03-20T14:58:22.286+0000 7fd500727700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-20T14:58:22.342 INFO:tasks.rook.operator.smithi007.stdout: stderr: [errno 2] RADOS object not found (error connecting to the cluster) 2022-03-20T14:58:22.343 INFO:tasks.rook.operator.smithi007.stdout:Traceback (most recent call last): 2022-03-20T14:58:22.343 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/sbin/ceph-volume", line 11, in 2022-03-20T14:58:22.343 INFO:tasks.rook.operator.smithi007.stdout: load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2022-03-20T14:58:22.343 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2022-03-20T14:58:22.343 INFO:tasks.rook.operator.smithi007.stdout: self.main(self.argv) 2022-03-20T14:58:22.344 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2022-03-20T14:58:22.344 INFO:tasks.rook.operator.smithi007.stdout: return f(*a, **kw) 2022-03-20T14:58:22.344 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2022-03-20T14:58:22.344 INFO:tasks.rook.operator.smithi007.stdout: terminal.dispatch(self.mapper, subcommand_args) 2022-03-20T14:58:22.345 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-20T14:58:22.345 INFO:tasks.rook.operator.smithi007.stdout: instance.main() 2022-03-20T14:58:22.345 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main 2022-03-20T14:58:22.345 INFO:tasks.rook.operator.smithi007.stdout: terminal.dispatch(self.mapper, self.argv) 2022-03-20T14:58:22.346 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-20T14:58:22.346 INFO:tasks.rook.operator.smithi007.stdout: instance.main() 2022-03-20T14:58:22.346 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main 2022-03-20T14:58:22.346 INFO:tasks.rook.operator.smithi007.stdout: self.safe_prepare(self.args) 2022-03-20T14:58:22.346 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-03-20T14:58:22.347 INFO:tasks.rook.operator.smithi007.stdout: self.prepare() 2022-03-20T14:58:22.347 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-03-20T14:58:22.347 INFO:tasks.rook.operator.smithi007.stdout: return func(*a, **kw) 2022-03-20T14:58:22.347 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-03-20T14:58:22.347 INFO:tasks.rook.operator.smithi007.stdout: osd_fsid, json.dumps(secrets)) 2022-03-20T14:58:22.348 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-03-20T14:58:22.348 INFO:tasks.rook.operator.smithi007.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-03-20T14:58:22.348 INFO:tasks.rook.operator.smithi007.stdout:RuntimeError: Unable to create a new OSD id. debug logs below: 2022-03-20T14:58:22.348 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,831][ceph_volume.main][INFO ] Running command: ceph-volume --log-path /var/log/ceph/smithi087-lv-3-data-0brh6f raw prepare --bluestore --data /dev/mapper/vg_nvme-lv_3 2022-03-20T14:58:22.349 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,832][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk -plno KNAME,NAME,TYPE 2022-03-20T14:58:22.349 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,836][ceph_volume.process][INFO ] stdout /dev/loop0 /dev/loop0 loop 2022-03-20T14:58:22.349 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,837][ceph_volume.process][INFO ] stdout /dev/loop1 /dev/loop1 loop 2022-03-20T14:58:22.349 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,837][ceph_volume.process][INFO ] stdout /dev/loop2 /dev/loop2 loop 2022-03-20T14:58:22.350 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,837][ceph_volume.process][INFO ] stdout /dev/loop3 /dev/loop3 loop 2022-03-20T14:58:22.350 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,837][ceph_volume.process][INFO ] stdout /dev/sda /dev/sda disk 2022-03-20T14:58:22.350 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,837][ceph_volume.process][INFO ] stdout /dev/sda1 /dev/sda1 part 2022-03-20T14:58:22.350 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,837][ceph_volume.process][INFO ] stdout /dev/dm-0 /dev/mapper/vg_nvme-lv_1 lvm 2022-03-20T14:58:22.350 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,837][ceph_volume.process][INFO ] stdout /dev/dm-1 /dev/mapper/vg_nvme-lv_2 lvm 2022-03-20T14:58:22.351 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,837][ceph_volume.process][INFO ] stdout /dev/dm-2 /dev/mapper/vg_nvme-lv_3 lvm 2022-03-20T14:58:22.351 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,837][ceph_volume.process][INFO ] stdout /dev/dm-3 /dev/mapper/vg_nvme-lv_4 lvm 2022-03-20T14:58:22.351 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,837][ceph_volume.process][INFO ] stdout /dev/dm-4 /dev/mapper/vg_nvme-lv_5 lvm 2022-03-20T14:58:22.351 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,837][ceph_volume.process][INFO ] stdout /dev/nvme0n1 /dev/nvme0n1 disk 2022-03-20T14:58:22.352 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,837][ceph_volume.process][INFO ] stdout /dev/nvme1n1 /dev/nvme1n1 disk 2022-03-20T14:58:22.352 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,837][ceph_volume.process][INFO ] stdout /dev/nvme2n1 /dev/nvme2n1 disk 2022-03-20T14:58:22.352 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,837][ceph_volume.process][INFO ] stdout /dev/nvme3n1 /dev/nvme3n1 disk 2022-03-20T14:58:22.352 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,837][ceph_volume.process][INFO ] stdout /dev/nvme4n1 /dev/nvme4n1 disk 2022-03-20T14:58:22.352 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,844][ceph_volume.util.system][WARNING] Executable lvs not found on the host, will return lvs as-is 2022-03-20T14:58:22.353 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,844][ceph_volume.process][INFO ] Running command: lvs --noheadings --readonly --separator=";" -a --units=b --nosuffix -S lv_path=/dev/mapper/vg_nvme-lv_3 -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size 2022-03-20T14:58:22.353 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,142][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk --nodeps -P -o NAME,KNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL /dev/mapper/vg_nvme-lv_3 2022-03-20T14:58:22.353 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,148][ceph_volume.process][INFO ] stdout NAME="vg_nvme-lv_3" KNAME="dm-2" MAJ:MIN="253:2" FSTYPE="" MOUNTPOINT="" LABEL="" UUID="" RO="0" RM="0" MODEL="" SIZE="89.4G" STATE="running" OWNER="root" GROUP="disk" MODE="brw-rw----" ALIGNMENT="0" PHY-SEC="512" LOG-SEC="512" ROTA="0" SCHED="" TYPE="lvm" DISC-ALN="0" DISC-GRAN="512B" DISC-MAX="2T" DISC-ZERO="0" PKNAME="" PARTLABEL="" 2022-03-20T14:58:22.353 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,149][ceph_volume.process][INFO ] Running command: /usr/sbin/blkid -c /dev/null -p /dev/mapper/vg_nvme-lv_3 2022-03-20T14:58:22.354 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,153][ceph_volume.util.disk][INFO ] opening device /dev/mapper/vg_nvme-lv_3 to check for BlueStore label 2022-03-20T14:58:22.354 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,153][ceph_volume.util.disk][INFO ] opening device /dev/mapper/vg_nvme-lv_3 to check for BlueStore label 2022-03-20T14:58:22.354 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,153][ceph_volume.process][INFO ] Running command: /usr/sbin/udevadm info --query=property /dev/mapper/vg_nvme-lv_3 2022-03-20T14:58:22.354 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,160][ceph_volume.process][INFO ] stdout DEVLINKS=/dev/vg_nvme/lv_3 /dev/disk/by-id/dm-uuid-LVM-UojCRVSNPvr1ajoeYEqVoya4dmuL8IkA0OYDTBQUXOqaBYoUpP9S4JZ1wtqOkqyL /dev/mapper/vg_nvme-lv_3 /dev/disk/by-id/dm-name-vg_nvme-lv_3 2022-03-20T14:58:22.355 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,160][ceph_volume.process][INFO ] stdout DEVNAME=/dev/dm-2 2022-03-20T14:58:22.355 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,160][ceph_volume.process][INFO ] stdout DEVPATH=/devices/virtual/block/dm-2 2022-03-20T14:58:22.355 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,160][ceph_volume.process][INFO ] stdout DEVTYPE=disk 2022-03-20T14:58:22.355 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,160][ceph_volume.process][INFO ] stdout DM_LV_NAME=lv_3 2022-03-20T14:58:22.355 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,160][ceph_volume.process][INFO ] stdout DM_NAME=vg_nvme-lv_3 2022-03-20T14:58:22.356 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,161][ceph_volume.process][INFO ] stdout DM_SUSPENDED=0 2022-03-20T14:58:22.356 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,161][ceph_volume.process][INFO ] stdout DM_UDEV_DISABLE_LIBRARY_FALLBACK_FLAG=1 2022-03-20T14:58:22.356 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,161][ceph_volume.process][INFO ] stdout DM_UDEV_PRIMARY_SOURCE_FLAG=1 2022-03-20T14:58:22.356 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,161][ceph_volume.process][INFO ] stdout DM_UDEV_RULES=1 2022-03-20T14:58:22.357 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,161][ceph_volume.process][INFO ] stdout DM_UDEV_RULES_VSN=2 2022-03-20T14:58:22.357 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,161][ceph_volume.process][INFO ] stdout DM_UUID=LVM-UojCRVSNPvr1ajoeYEqVoya4dmuL8IkA0OYDTBQUXOqaBYoUpP9S4JZ1wtqOkqyL 2022-03-20T14:58:22.357 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,161][ceph_volume.process][INFO ] stdout DM_VG_NAME=vg_nvme 2022-03-20T14:58:22.357 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,161][ceph_volume.process][INFO ] stdout MAJOR=253 2022-03-20T14:58:22.357 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,161][ceph_volume.process][INFO ] stdout MINOR=2 2022-03-20T14:58:22.358 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,161][ceph_volume.process][INFO ] stdout SUBSYSTEM=block 2022-03-20T14:58:22.358 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,161][ceph_volume.process][INFO ] stdout TAGS=:systemd: 2022-03-20T14:58:22.358 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,161][ceph_volume.process][INFO ] stdout USEC_INITIALIZED=220200660 2022-03-20T14:58:22.358 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,162][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-authtool --gen-print-key 2022-03-20T14:58:22.359 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,181][ceph_volume.process][INFO ] stdout AQAOQTdihIPAChAA+/JfmkBufyKYFVCUYP52Zw== 2022-03-20T14:58:22.359 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,182][ceph_volume.process][INFO ] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e1856f16-99e2-4716-b7b5-bcf6c397f73d 2022-03-20T14:58:22.359 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,293][ceph_volume.process][INFO ] stderr unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-20T14:58:22.359 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,293][ceph_volume.process][INFO ] stderr 2022-03-20T14:58:22.286+0000 7fd500727700 -1 failed for service _ceph-mon._tcp 2022-03-20T14:58:22.360 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,293][ceph_volume.process][INFO ] stderr 2022-03-20T14:58:22.286+0000 7fd500727700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-20T14:58:22.360 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,293][ceph_volume.process][INFO ] stderr [errno 2] RADOS object not found (error connecting to the cluster) 2022-03-20T14:58:22.360 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,293][ceph_volume.devices.raw.prepare][ERROR ] raw prepare was unable to complete 2022-03-20T14:58:22.360 INFO:tasks.rook.operator.smithi007.stdout:Traceback (most recent call last): 2022-03-20T14:58:22.361 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-03-20T14:58:22.361 INFO:tasks.rook.operator.smithi007.stdout: self.prepare() 2022-03-20T14:58:22.361 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-03-20T14:58:22.361 INFO:tasks.rook.operator.smithi007.stdout: return func(*a, **kw) 2022-03-20T14:58:22.362 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-03-20T14:58:22.362 INFO:tasks.rook.operator.smithi007.stdout: osd_fsid, json.dumps(secrets)) 2022-03-20T14:58:22.362 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-03-20T14:58:22.362 INFO:tasks.rook.operator.smithi007.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-03-20T14:58:22.362 INFO:tasks.rook.operator.smithi007.stdout:RuntimeError: Unable to create a new OSD id 2022-03-20T14:58:22.363 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,294][ceph_volume.devices.raw.prepare][INFO ] will rollback OSD ID creation 2022-03-20T14:58:22.363 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,294][ceph_volume][ERROR ] exception caught by decorator 2022-03-20T14:58:22.363 INFO:tasks.rook.operator.smithi007.stdout:Traceback (most recent call last): 2022-03-20T14:58:22.363 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2022-03-20T14:58:22.364 INFO:tasks.rook.operator.smithi007.stdout: return f(*a, **kw) 2022-03-20T14:58:22.364 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2022-03-20T14:58:22.364 INFO:tasks.rook.operator.smithi007.stdout: terminal.dispatch(self.mapper, subcommand_args) 2022-03-20T14:58:22.364 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-20T14:58:22.365 INFO:tasks.rook.operator.smithi007.stdout: instance.main() 2022-03-20T14:58:22.365 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main 2022-03-20T14:58:22.365 INFO:tasks.rook.operator.smithi007.stdout: terminal.dispatch(self.mapper, self.argv) 2022-03-20T14:58:22.365 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-20T14:58:22.365 INFO:tasks.rook.operator.smithi007.stdout: instance.main() 2022-03-20T14:58:22.366 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main 2022-03-20T14:58:22.366 INFO:tasks.rook.operator.smithi007.stdout: self.safe_prepare(self.args) 2022-03-20T14:58:22.366 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-03-20T14:58:22.366 INFO:tasks.rook.operator.smithi007.stdout: self.prepare() 2022-03-20T14:58:22.367 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-03-20T14:58:22.367 INFO:tasks.rook.operator.smithi007.stdout: return func(*a, **kw) 2022-03-20T14:58:22.367 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-03-20T14:58:22.367 INFO:tasks.rook.operator.smithi007.stdout: osd_fsid, json.dumps(secrets)) 2022-03-20T14:58:22.368 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-03-20T14:58:22.368 INFO:tasks.rook.operator.smithi007.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-03-20T14:58:22.368 INFO:tasks.rook.operator.smithi007.stdout:RuntimeError: Unable to create a new OSD id 2022-03-20T14:58:22.368 INFO:tasks.rook.operator.smithi007.stdout:: exit status 1} 2022-03-20T14:58:22.458 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:22.457771 I | op-osd: OSD orchestration status for PVC smithi087-lv-2-data-04lndq is "failed" 2022-03-20T14:58:22.459 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:22.457812 E | op-osd: failed to provision OSD(s) on PVC smithi087-lv-2-data-04lndq. &{OSDs:[] Status:failed PvcBackedOSD:true Message:failed to configure devices: failed to initialize devices on PVC: failed to run ceph-volume. Running command: /usr/bin/ceph-authtool --gen-print-key 2022-03-20T14:58:22.459 INFO:tasks.rook.operator.smithi007.stdout:Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 198569b9-1e07-4d16-a236-026a4595652c 2022-03-20T14:58:22.459 INFO:tasks.rook.operator.smithi007.stdout: stderr: unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-20T14:58:22.460 INFO:tasks.rook.operator.smithi007.stdout: stderr: 2022-03-20T14:58:22.294+0000 7f8ea846d700 -1 failed for service _ceph-mon._tcp 2022-03-20T14:58:22.460 INFO:tasks.rook.operator.smithi007.stdout: stderr: 2022-03-20T14:58:22.294+0000 7f8ea846d700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-20T14:58:22.460 INFO:tasks.rook.operator.smithi007.stdout: stderr: [errno 2] RADOS object not found (error connecting to the cluster) 2022-03-20T14:58:22.461 INFO:tasks.rook.operator.smithi007.stdout:Traceback (most recent call last): 2022-03-20T14:58:22.461 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/sbin/ceph-volume", line 11, in 2022-03-20T14:58:22.461 INFO:tasks.rook.operator.smithi007.stdout: load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2022-03-20T14:58:22.461 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2022-03-20T14:58:22.462 INFO:tasks.rook.operator.smithi007.stdout: self.main(self.argv) 2022-03-20T14:58:22.462 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2022-03-20T14:58:22.462 INFO:tasks.rook.operator.smithi007.stdout: return f(*a, **kw) 2022-03-20T14:58:22.463 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2022-03-20T14:58:22.463 INFO:tasks.rook.operator.smithi007.stdout: terminal.dispatch(self.mapper, subcommand_args) 2022-03-20T14:58:22.463 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-20T14:58:22.464 INFO:tasks.rook.operator.smithi007.stdout: instance.main() 2022-03-20T14:58:22.464 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main 2022-03-20T14:58:22.464 INFO:tasks.rook.operator.smithi007.stdout: terminal.dispatch(self.mapper, self.argv) 2022-03-20T14:58:22.465 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-20T14:58:22.465 INFO:tasks.rook.operator.smithi007.stdout: instance.main() 2022-03-20T14:58:22.465 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main 2022-03-20T14:58:22.465 INFO:tasks.rook.operator.smithi007.stdout: self.safe_prepare(self.args) 2022-03-20T14:58:22.466 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-03-20T14:58:22.466 INFO:tasks.rook.operator.smithi007.stdout: self.prepare() 2022-03-20T14:58:22.466 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-03-20T14:58:22.467 INFO:tasks.rook.operator.smithi007.stdout: return func(*a, **kw) 2022-03-20T14:58:22.467 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-03-20T14:58:22.467 INFO:tasks.rook.operator.smithi007.stdout: osd_fsid, json.dumps(secrets)) 2022-03-20T14:58:22.468 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-03-20T14:58:22.468 INFO:tasks.rook.operator.smithi007.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-03-20T14:58:22.479 INFO:tasks.rook.operator.smithi007.stdout:RuntimeError: Unable to create a new OSD id. debug logs below: 2022-03-20T14:58:22.479 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,837][ceph_volume.main][INFO ] Running command: ceph-volume --log-path /var/log/ceph/smithi087-lv-2-data-04lndq raw prepare --bluestore --data /dev/mapper/vg_nvme-lv_2 2022-03-20T14:58:22.479 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,839][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk -plno KNAME,NAME,TYPE 2022-03-20T14:58:22.480 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,845][ceph_volume.process][INFO ] stdout /dev/loop0 /dev/loop0 loop 2022-03-20T14:58:22.480 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,845][ceph_volume.process][INFO ] stdout /dev/loop1 /dev/loop1 loop 2022-03-20T14:58:22.480 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,845][ceph_volume.process][INFO ] stdout /dev/loop2 /dev/loop2 loop 2022-03-20T14:58:22.480 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,845][ceph_volume.process][INFO ] stdout /dev/loop3 /dev/loop3 loop 2022-03-20T14:58:22.481 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,845][ceph_volume.process][INFO ] stdout /dev/sda /dev/sda disk 2022-03-20T14:58:22.481 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,845][ceph_volume.process][INFO ] stdout /dev/sda1 /dev/sda1 part 2022-03-20T14:58:22.481 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,845][ceph_volume.process][INFO ] stdout /dev/dm-0 /dev/mapper/vg_nvme-lv_1 lvm 2022-03-20T14:58:22.481 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,845][ceph_volume.process][INFO ] stdout /dev/dm-1 /dev/mapper/vg_nvme-lv_2 lvm 2022-03-20T14:58:22.481 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,845][ceph_volume.process][INFO ] stdout /dev/dm-2 /dev/mapper/vg_nvme-lv_3 lvm 2022-03-20T14:58:22.482 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,845][ceph_volume.process][INFO ] stdout /dev/dm-3 /dev/mapper/vg_nvme-lv_4 lvm 2022-03-20T14:58:22.482 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,845][ceph_volume.process][INFO ] stdout /dev/dm-4 /dev/mapper/vg_nvme-lv_5 lvm 2022-03-20T14:58:22.482 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,845][ceph_volume.process][INFO ] stdout /dev/nvme0n1 /dev/nvme0n1 disk 2022-03-20T14:58:22.482 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,845][ceph_volume.process][INFO ] stdout /dev/nvme1n1 /dev/nvme1n1 disk 2022-03-20T14:58:22.483 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,845][ceph_volume.process][INFO ] stdout /dev/nvme2n1 /dev/nvme2n1 disk 2022-03-20T14:58:22.483 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,846][ceph_volume.process][INFO ] stdout /dev/nvme3n1 /dev/nvme3n1 disk 2022-03-20T14:58:22.483 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,846][ceph_volume.process][INFO ] stdout /dev/nvme4n1 /dev/nvme4n1 disk 2022-03-20T14:58:22.483 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,856][ceph_volume.util.system][WARNING] Executable lvs not found on the host, will return lvs as-is 2022-03-20T14:58:22.483 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,856][ceph_volume.process][INFO ] Running command: lvs --noheadings --readonly --separator=";" -a --units=b --nosuffix -S lv_path=/dev/mapper/vg_nvme-lv_2 -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size 2022-03-20T14:58:22.484 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,157][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk --nodeps -P -o NAME,KNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL /dev/mapper/vg_nvme-lv_2 2022-03-20T14:58:22.484 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,163][ceph_volume.process][INFO ] stdout NAME="vg_nvme-lv_2" KNAME="dm-1" MAJ:MIN="253:1" FSTYPE="" MOUNTPOINT="" LABEL="" UUID="" RO="0" RM="0" MODEL="" SIZE="89.4G" STATE="running" OWNER="root" GROUP="disk" MODE="brw-rw----" ALIGNMENT="0" PHY-SEC="512" LOG-SEC="512" ROTA="0" SCHED="" TYPE="lvm" DISC-ALN="0" DISC-GRAN="512B" DISC-MAX="2T" DISC-ZERO="0" PKNAME="" PARTLABEL="" 2022-03-20T14:58:22.484 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,163][ceph_volume.process][INFO ] Running command: /usr/sbin/blkid -c /dev/null -p /dev/mapper/vg_nvme-lv_2 2022-03-20T14:58:22.484 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,166][ceph_volume.util.disk][INFO ] opening device /dev/mapper/vg_nvme-lv_2 to check for BlueStore label 2022-03-20T14:58:22.485 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,167][ceph_volume.util.disk][INFO ] opening device /dev/mapper/vg_nvme-lv_2 to check for BlueStore label 2022-03-20T14:58:22.485 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,167][ceph_volume.process][INFO ] Running command: /usr/sbin/udevadm info --query=property /dev/mapper/vg_nvme-lv_2 2022-03-20T14:58:22.485 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,173][ceph_volume.process][INFO ] stdout DEVLINKS=/dev/disk/by-id/dm-uuid-LVM-UojCRVSNPvr1ajoeYEqVoya4dmuL8IkAYkwQoadiqgTuV3p0HiOt9Ek1foK2kJ2t /dev/vg_nvme/lv_2 /dev/mapper/vg_nvme-lv_2 /dev/disk/by-id/dm-name-vg_nvme-lv_2 2022-03-20T14:58:22.485 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,174][ceph_volume.process][INFO ] stdout DEVNAME=/dev/dm-1 2022-03-20T14:58:22.485 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,174][ceph_volume.process][INFO ] stdout DEVPATH=/devices/virtual/block/dm-1 2022-03-20T14:58:22.486 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,174][ceph_volume.process][INFO ] stdout DEVTYPE=disk 2022-03-20T14:58:22.486 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,174][ceph_volume.process][INFO ] stdout DM_LV_NAME=lv_2 2022-03-20T14:58:22.486 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,174][ceph_volume.process][INFO ] stdout DM_NAME=vg_nvme-lv_2 2022-03-20T14:58:22.486 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,174][ceph_volume.process][INFO ] stdout DM_SUSPENDED=0 2022-03-20T14:58:22.487 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,174][ceph_volume.process][INFO ] stdout DM_UDEV_DISABLE_LIBRARY_FALLBACK_FLAG=1 2022-03-20T14:58:22.487 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,174][ceph_volume.process][INFO ] stdout DM_UDEV_PRIMARY_SOURCE_FLAG=1 2022-03-20T14:58:22.487 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,174][ceph_volume.process][INFO ] stdout DM_UDEV_RULES=1 2022-03-20T14:58:22.487 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,174][ceph_volume.process][INFO ] stdout DM_UDEV_RULES_VSN=2 2022-03-20T14:58:22.487 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,174][ceph_volume.process][INFO ] stdout DM_UUID=LVM-UojCRVSNPvr1ajoeYEqVoya4dmuL8IkAYkwQoadiqgTuV3p0HiOt9Ek1foK2kJ2t 2022-03-20T14:58:22.488 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,174][ceph_volume.process][INFO ] stdout DM_VG_NAME=vg_nvme 2022-03-20T14:58:22.488 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,174][ceph_volume.process][INFO ] stdout MAJOR=253 2022-03-20T14:58:22.488 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,174][ceph_volume.process][INFO ] stdout MINOR=1 2022-03-20T14:58:22.488 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,174][ceph_volume.process][INFO ] stdout SUBSYSTEM=block 2022-03-20T14:58:22.489 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,175][ceph_volume.process][INFO ] stdout TAGS=:systemd: 2022-03-20T14:58:22.489 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,175][ceph_volume.process][INFO ] stdout USEC_INITIALIZED=219766073 2022-03-20T14:58:22.489 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,175][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-authtool --gen-print-key 2022-03-20T14:58:22.489 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,191][ceph_volume.process][INFO ] stdout AQAOQTdix/tYCxAAFwamna54d6Ua2fcs5vYbgA== 2022-03-20T14:58:22.489 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,192][ceph_volume.process][INFO ] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 198569b9-1e07-4d16-a236-026a4595652c 2022-03-20T14:58:22.490 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,299][ceph_volume.process][INFO ] stderr unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-20T14:58:22.490 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,299][ceph_volume.process][INFO ] stderr 2022-03-20T14:58:22.294+0000 7f8ea846d700 -1 failed for service _ceph-mon._tcp 2022-03-20T14:58:22.490 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,299][ceph_volume.process][INFO ] stderr 2022-03-20T14:58:22.294+0000 7f8ea846d700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-20T14:58:22.490 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,299][ceph_volume.process][INFO ] stderr [errno 2] RADOS object not found (error connecting to the cluster) 2022-03-20T14:58:22.491 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,299][ceph_volume.devices.raw.prepare][ERROR ] raw prepare was unable to complete 2022-03-20T14:58:22.491 INFO:tasks.rook.operator.smithi007.stdout:Traceback (most recent call last): 2022-03-20T14:58:22.491 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-03-20T14:58:22.491 INFO:tasks.rook.operator.smithi007.stdout: self.prepare() 2022-03-20T14:58:22.492 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-03-20T14:58:22.492 INFO:tasks.rook.operator.smithi007.stdout: return func(*a, **kw) 2022-03-20T14:58:22.492 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-03-20T14:58:22.493 INFO:tasks.rook.operator.smithi007.stdout: osd_fsid, json.dumps(secrets)) 2022-03-20T14:58:22.493 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-03-20T14:58:22.493 INFO:tasks.rook.operator.smithi007.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-03-20T14:58:22.493 INFO:tasks.rook.operator.smithi007.stdout:RuntimeError: Unable to create a new OSD id 2022-03-20T14:58:22.493 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,300][ceph_volume.devices.raw.prepare][INFO ] will rollback OSD ID creation 2022-03-20T14:58:22.494 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,300][ceph_volume][ERROR ] exception caught by decorator 2022-03-20T14:58:22.494 INFO:tasks.rook.operator.smithi007.stdout:Traceback (most recent call last): 2022-03-20T14:58:22.494 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2022-03-20T14:58:22.494 INFO:tasks.rook.operator.smithi007.stdout: return f(*a, **kw) 2022-03-20T14:58:22.495 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2022-03-20T14:58:22.495 INFO:tasks.rook.operator.smithi007.stdout: terminal.dispatch(self.mapper, subcommand_args) 2022-03-20T14:58:22.495 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-20T14:58:22.495 INFO:tasks.rook.operator.smithi007.stdout: instance.main() 2022-03-20T14:58:22.495 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main 2022-03-20T14:58:22.496 INFO:tasks.rook.operator.smithi007.stdout: terminal.dispatch(self.mapper, self.argv) 2022-03-20T14:58:22.496 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-20T14:58:22.496 INFO:tasks.rook.operator.smithi007.stdout: instance.main() 2022-03-20T14:58:22.496 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main 2022-03-20T14:58:22.497 INFO:tasks.rook.operator.smithi007.stdout: self.safe_prepare(self.args) 2022-03-20T14:58:22.497 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-03-20T14:58:22.497 INFO:tasks.rook.operator.smithi007.stdout: self.prepare() 2022-03-20T14:58:22.497 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-03-20T14:58:22.497 INFO:tasks.rook.operator.smithi007.stdout: return func(*a, **kw) 2022-03-20T14:58:22.498 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-03-20T14:58:22.498 INFO:tasks.rook.operator.smithi007.stdout: osd_fsid, json.dumps(secrets)) 2022-03-20T14:58:22.498 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-03-20T14:58:22.498 INFO:tasks.rook.operator.smithi007.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-03-20T14:58:22.499 INFO:tasks.rook.operator.smithi007.stdout:RuntimeError: Unable to create a new OSD id 2022-03-20T14:58:22.499 INFO:tasks.rook.operator.smithi007.stdout:: exit status 1} 2022-03-20T14:58:24.828 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:24.832296 I | op-osd: OSD orchestration status for PVC smithi057-lv-2-data-0xdlsb is "completed" 2022-03-20T14:58:24.828 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:24.832333 I | op-osd: creating OSD 0 on PVC "smithi057-lv-2-data-0xdlsb" 2022-03-20T14:58:24.829 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:24.832347 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-2-data-0xdlsb" 2022-03-20T14:58:25.270 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:25.266203 I | op-osd: OSD orchestration status for node smithi087-lv-2-data-04lndq is "orchestrating" 2022-03-20T14:58:25.276 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:25.280356 I | op-osd: OSD orchestration status for PVC smithi057-lv-4-data-0zt8b4 is "completed" 2022-03-20T14:58:25.276 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:25.280378 I | op-osd: creating OSD 1 on PVC "smithi057-lv-4-data-0zt8b4" 2022-03-20T14:58:25.277 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:25.280394 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-4-data-0zt8b4" 2022-03-20T14:58:25.437 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:25.441751 I | op-osd: OSD orchestration status for node smithi087-lv-3-data-0brh6f is "orchestrating" 2022-03-20T14:58:25.793 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:25.797961 I | op-osd: OSD orchestration status for PVC smithi087-lv-3-data-0brh6f is "orchestrating" 2022-03-20T14:58:25.794 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:25.797986 I | op-osd: OSD orchestration status for PVC smithi087-lv-2-data-04lndq is "orchestrating" 2022-03-20T14:58:25.946 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd stat -f json 2022-03-20T14:58:26.042 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:26.045642 I | op-osd: OSD orchestration status for PVC smithi057-lv-1-data-056qwk is "completed" 2022-03-20T14:58:26.042 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:26.045662 I | op-osd: creating OSD 2 on PVC "smithi057-lv-1-data-056qwk" 2022-03-20T14:58:26.043 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:26.045669 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-1-data-056qwk" 2022-03-20T14:58:26.247 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:26.250768 I | op-osd: OSD orchestration status for PVC smithi087-lv-1-data-0p7bb2 is "completed" 2022-03-20T14:58:26.247 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:26.250786 I | op-osd: creating OSD 4 on PVC "smithi087-lv-1-data-0p7bb2" 2022-03-20T14:58:26.248 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:26.250794 I | op-osd: OSD will have its main bluestore block on "smithi087-lv-1-data-0p7bb2" 2022-03-20T14:58:26.670 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:26.674561 I | op-osd: OSD orchestration status for PVC smithi057-lv-3-data-0vrztn is "completed" 2022-03-20T14:58:26.671 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:26.674581 I | op-osd: creating OSD 3 on PVC "smithi057-lv-3-data-0vrztn" 2022-03-20T14:58:26.671 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:26.674588 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-3-data-0vrztn" 2022-03-20T14:58:27.215 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T14:58:27.243 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":12,"num_osds":10,"num_up_osds":0,"osd_up_since":0,"num_in_osds":10,"osd_in_since":1647788303,"num_remapped_pgs":0} 2022-03-20T14:58:27.243 INFO:tasks.rook: have 0/12 OSDs 2022-03-20T14:58:27.614 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:27.618311 I | op-osd: OSD orchestration status for PVC smithi007-lv-4-data-0snpfz is "completed" 2022-03-20T14:58:27.615 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:27.618331 I | op-osd: creating OSD 5 on PVC "smithi007-lv-4-data-0snpfz" 2022-03-20T14:58:27.615 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:27.618338 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-4-data-0snpfz" 2022-03-20T14:58:28.430 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:28.434131 I | op-osd: OSD orchestration status for PVC smithi007-lv-1-data-0fltgg is "completed" 2022-03-20T14:58:28.431 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:28.434156 I | op-osd: creating OSD 7 on PVC "smithi007-lv-1-data-0fltgg" 2022-03-20T14:58:28.431 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:28.434166 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-1-data-0fltgg" 2022-03-20T14:58:31.531 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:31.532609 I | op-osd: OSD orchestration status for PVC smithi087-lv-4-data-0vssdn is "completed" 2022-03-20T14:58:31.531 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:31.532631 I | op-osd: creating OSD 6 on PVC "smithi087-lv-4-data-0vssdn" 2022-03-20T14:58:31.531 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:31.532642 I | op-osd: OSD will have its main bluestore block on "smithi087-lv-4-data-0vssdn" 2022-03-20T14:58:35.004 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:35.007363 I | op-osd: OSD orchestration status for PVC smithi007-lv-3-data-0wb6kn is "completed" 2022-03-20T14:58:35.004 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:35.007422 I | op-osd: creating OSD 8 on PVC "smithi007-lv-3-data-0wb6kn" 2022-03-20T14:58:35.004 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:35.007434 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-3-data-0wb6kn" 2022-03-20T14:58:36.462 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:36.463103 I | op-osd: OSD orchestration status for PVC smithi007-lv-2-data-0z6lbn is "completed" 2022-03-20T14:58:36.463 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:36.463159 I | op-osd: creating OSD 9 on PVC "smithi007-lv-2-data-0z6lbn" 2022-03-20T14:58:36.463 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:36.463244 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-2-data-0z6lbn" 2022-03-20T14:58:37.245 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd stat -f json 2022-03-20T14:58:39.188 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T14:58:39.206 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":14,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1647788308,"num_remapped_pgs":0} 2022-03-20T14:58:39.207 INFO:tasks.rook: have 0/12 OSDs 2022-03-20T14:58:39.541 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:39.544394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph osds: 2 failures encountered while running osds on nodes in namespace "rook-ceph". 2022-03-20T14:58:39.541 INFO:tasks.rook.operator.smithi007.stdout:failed to provision OSD(s) on PVC smithi087-lv-3-data-0brh6f. &{OSDs:[] Status:failed PvcBackedOSD:true Message:failed to configure devices: failed to initialize devices on PVC: failed to run ceph-volume. Running command: /usr/bin/ceph-authtool --gen-print-key 2022-03-20T14:58:39.542 INFO:tasks.rook.operator.smithi007.stdout:Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e1856f16-99e2-4716-b7b5-bcf6c397f73d 2022-03-20T14:58:39.542 INFO:tasks.rook.operator.smithi007.stdout: stderr: unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-20T14:58:39.542 INFO:tasks.rook.operator.smithi007.stdout: stderr: 2022-03-20T14:58:22.286+0000 7fd500727700 -1 failed for service _ceph-mon._tcp 2022-03-20T14:58:39.542 INFO:tasks.rook.operator.smithi007.stdout: stderr: 2022-03-20T14:58:22.286+0000 7fd500727700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-20T14:58:39.543 INFO:tasks.rook.operator.smithi007.stdout: stderr: [errno 2] RADOS object not found (error connecting to the cluster) 2022-03-20T14:58:39.543 INFO:tasks.rook.operator.smithi007.stdout:Traceback (most recent call last): 2022-03-20T14:58:39.543 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/sbin/ceph-volume", line 11, in 2022-03-20T14:58:39.543 INFO:tasks.rook.operator.smithi007.stdout: load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2022-03-20T14:58:39.543 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2022-03-20T14:58:39.544 INFO:tasks.rook.operator.smithi007.stdout: self.main(self.argv) 2022-03-20T14:58:39.544 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2022-03-20T14:58:39.544 INFO:tasks.rook.operator.smithi007.stdout: return f(*a, **kw) 2022-03-20T14:58:39.544 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2022-03-20T14:58:39.545 INFO:tasks.rook.operator.smithi007.stdout: terminal.dispatch(self.mapper, subcommand_args) 2022-03-20T14:58:39.545 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-20T14:58:39.545 INFO:tasks.rook.operator.smithi007.stdout: instance.main() 2022-03-20T14:58:39.545 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main 2022-03-20T14:58:39.545 INFO:tasks.rook.operator.smithi007.stdout: terminal.dispatch(self.mapper, self.argv) 2022-03-20T14:58:39.546 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-20T14:58:39.546 INFO:tasks.rook.operator.smithi007.stdout: instance.main() 2022-03-20T14:58:39.546 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main 2022-03-20T14:58:39.546 INFO:tasks.rook.operator.smithi007.stdout: self.safe_prepare(self.args) 2022-03-20T14:58:39.546 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-03-20T14:58:39.561 INFO:tasks.rook.operator.smithi007.stdout: self.prepare() 2022-03-20T14:58:39.561 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-03-20T14:58:39.561 INFO:tasks.rook.operator.smithi007.stdout: return func(*a, **kw) 2022-03-20T14:58:39.562 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-03-20T14:58:39.562 INFO:tasks.rook.operator.smithi007.stdout: osd_fsid, json.dumps(secrets)) 2022-03-20T14:58:39.562 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-03-20T14:58:39.562 INFO:tasks.rook.operator.smithi007.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-03-20T14:58:39.563 INFO:tasks.rook.operator.smithi007.stdout:RuntimeError: Unable to create a new OSD id. debug logs below: 2022-03-20T14:58:39.563 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,831][ceph_volume.main][INFO ] Running command: ceph-volume --log-path /var/log/ceph/smithi087-lv-3-data-0brh6f raw prepare --bluestore --data /dev/mapper/vg_nvme-lv_3 2022-03-20T14:58:39.563 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,832][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk -plno KNAME,NAME,TYPE 2022-03-20T14:58:39.563 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,836][ceph_volume.process][INFO ] stdout /dev/loop0 /dev/loop0 loop 2022-03-20T14:58:39.564 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,837][ceph_volume.process][INFO ] stdout /dev/loop1 /dev/loop1 loop 2022-03-20T14:58:39.564 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,837][ceph_volume.process][INFO ] stdout /dev/loop2 /dev/loop2 loop 2022-03-20T14:58:39.564 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,837][ceph_volume.process][INFO ] stdout /dev/loop3 /dev/loop3 loop 2022-03-20T14:58:39.564 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,837][ceph_volume.process][INFO ] stdout /dev/sda /dev/sda disk 2022-03-20T14:58:39.564 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,837][ceph_volume.process][INFO ] stdout /dev/sda1 /dev/sda1 part 2022-03-20T14:58:39.565 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,837][ceph_volume.process][INFO ] stdout /dev/dm-0 /dev/mapper/vg_nvme-lv_1 lvm 2022-03-20T14:58:39.565 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,837][ceph_volume.process][INFO ] stdout /dev/dm-1 /dev/mapper/vg_nvme-lv_2 lvm 2022-03-20T14:58:39.565 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,837][ceph_volume.process][INFO ] stdout /dev/dm-2 /dev/mapper/vg_nvme-lv_3 lvm 2022-03-20T14:58:39.565 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,837][ceph_volume.process][INFO ] stdout /dev/dm-3 /dev/mapper/vg_nvme-lv_4 lvm 2022-03-20T14:58:39.565 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,837][ceph_volume.process][INFO ] stdout /dev/dm-4 /dev/mapper/vg_nvme-lv_5 lvm 2022-03-20T14:58:39.566 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,837][ceph_volume.process][INFO ] stdout /dev/nvme0n1 /dev/nvme0n1 disk 2022-03-20T14:58:39.566 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,837][ceph_volume.process][INFO ] stdout /dev/nvme1n1 /dev/nvme1n1 disk 2022-03-20T14:58:39.566 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,837][ceph_volume.process][INFO ] stdout /dev/nvme2n1 /dev/nvme2n1 disk 2022-03-20T14:58:39.566 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,837][ceph_volume.process][INFO ] stdout /dev/nvme3n1 /dev/nvme3n1 disk 2022-03-20T14:58:39.566 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,837][ceph_volume.process][INFO ] stdout /dev/nvme4n1 /dev/nvme4n1 disk 2022-03-20T14:58:39.567 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,844][ceph_volume.util.system][WARNING] Executable lvs not found on the host, will return lvs as-is 2022-03-20T14:58:39.567 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,844][ceph_volume.process][INFO ] Running command: lvs --noheadings --readonly --separator=";" -a --units=b --nosuffix -S lv_path=/dev/mapper/vg_nvme-lv_3 -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size 2022-03-20T14:58:39.567 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,142][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk --nodeps -P -o NAME,KNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL /dev/mapper/vg_nvme-lv_3 2022-03-20T14:58:39.567 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,148][ceph_volume.process][INFO ] stdout NAME="vg_nvme-lv_3" KNAME="dm-2" MAJ:MIN="253:2" FSTYPE="" MOUNTPOINT="" LABEL="" UUID="" RO="0" RM="0" MODEL="" SIZE="89.4G" STATE="running" OWNER="root" GROUP="disk" MODE="brw-rw----" ALIGNMENT="0" PHY-SEC="512" LOG-SEC="512" ROTA="0" SCHED="" TYPE="lvm" DISC-ALN="0" DISC-GRAN="512B" DISC-MAX="2T" DISC-ZERO="0" PKNAME="" PARTLABEL="" 2022-03-20T14:58:39.567 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,149][ceph_volume.process][INFO ] Running command: /usr/sbin/blkid -c /dev/null -p /dev/mapper/vg_nvme-lv_3 2022-03-20T14:58:39.568 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,153][ceph_volume.util.disk][INFO ] opening device /dev/mapper/vg_nvme-lv_3 to check for BlueStore label 2022-03-20T14:58:39.568 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,153][ceph_volume.util.disk][INFO ] opening device /dev/mapper/vg_nvme-lv_3 to check for BlueStore label 2022-03-20T14:58:39.568 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,153][ceph_volume.process][INFO ] Running command: /usr/sbin/udevadm info --query=property /dev/mapper/vg_nvme-lv_3 2022-03-20T14:58:39.568 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,160][ceph_volume.process][INFO ] stdout DEVLINKS=/dev/vg_nvme/lv_3 /dev/disk/by-id/dm-uuid-LVM-UojCRVSNPvr1ajoeYEqVoya4dmuL8IkA0OYDTBQUXOqaBYoUpP9S4JZ1wtqOkqyL /dev/mapper/vg_nvme-lv_3 /dev/disk/by-id/dm-name-vg_nvme-lv_3 2022-03-20T14:58:39.568 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,160][ceph_volume.process][INFO ] stdout DEVNAME=/dev/dm-2 2022-03-20T14:58:39.569 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,160][ceph_volume.process][INFO ] stdout DEVPATH=/devices/virtual/block/dm-2 2022-03-20T14:58:39.569 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,160][ceph_volume.process][INFO ] stdout DEVTYPE=disk 2022-03-20T14:58:39.569 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,160][ceph_volume.process][INFO ] stdout DM_LV_NAME=lv_3 2022-03-20T14:58:39.569 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,160][ceph_volume.process][INFO ] stdout DM_NAME=vg_nvme-lv_3 2022-03-20T14:58:39.569 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,161][ceph_volume.process][INFO ] stdout DM_SUSPENDED=0 2022-03-20T14:58:39.570 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,161][ceph_volume.process][INFO ] stdout DM_UDEV_DISABLE_LIBRARY_FALLBACK_FLAG=1 2022-03-20T14:58:39.570 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,161][ceph_volume.process][INFO ] stdout DM_UDEV_PRIMARY_SOURCE_FLAG=1 2022-03-20T14:58:39.570 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,161][ceph_volume.process][INFO ] stdout DM_UDEV_RULES=1 2022-03-20T14:58:39.570 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,161][ceph_volume.process][INFO ] stdout DM_UDEV_RULES_VSN=2 2022-03-20T14:58:39.571 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,161][ceph_volume.process][INFO ] stdout DM_UUID=LVM-UojCRVSNPvr1ajoeYEqVoya4dmuL8IkA0OYDTBQUXOqaBYoUpP9S4JZ1wtqOkqyL 2022-03-20T14:58:39.571 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,161][ceph_volume.process][INFO ] stdout DM_VG_NAME=vg_nvme 2022-03-20T14:58:39.571 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,161][ceph_volume.process][INFO ] stdout MAJOR=253 2022-03-20T14:58:39.571 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,161][ceph_volume.process][INFO ] stdout MINOR=2 2022-03-20T14:58:39.571 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,161][ceph_volume.process][INFO ] stdout SUBSYSTEM=block 2022-03-20T14:58:39.572 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,161][ceph_volume.process][INFO ] stdout TAGS=:systemd: 2022-03-20T14:58:39.572 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,161][ceph_volume.process][INFO ] stdout USEC_INITIALIZED=220200660 2022-03-20T14:58:39.572 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,162][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-authtool --gen-print-key 2022-03-20T14:58:39.572 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,181][ceph_volume.process][INFO ] stdout AQAOQTdihIPAChAA+/JfmkBufyKYFVCUYP52Zw== 2022-03-20T14:58:39.572 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,182][ceph_volume.process][INFO ] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e1856f16-99e2-4716-b7b5-bcf6c397f73d 2022-03-20T14:58:39.573 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,293][ceph_volume.process][INFO ] stderr unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-20T14:58:39.573 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,293][ceph_volume.process][INFO ] stderr 2022-03-20T14:58:22.286+0000 7fd500727700 -1 failed for service _ceph-mon._tcp 2022-03-20T14:58:39.573 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,293][ceph_volume.process][INFO ] stderr 2022-03-20T14:58:22.286+0000 7fd500727700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-20T14:58:39.573 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,293][ceph_volume.process][INFO ] stderr [errno 2] RADOS object not found (error connecting to the cluster) 2022-03-20T14:58:39.573 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,293][ceph_volume.devices.raw.prepare][ERROR ] raw prepare was unable to complete 2022-03-20T14:58:39.574 INFO:tasks.rook.operator.smithi007.stdout:Traceback (most recent call last): 2022-03-20T14:58:39.574 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-03-20T14:58:39.574 INFO:tasks.rook.operator.smithi007.stdout: self.prepare() 2022-03-20T14:58:39.574 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-03-20T14:58:39.574 INFO:tasks.rook.operator.smithi007.stdout: return func(*a, **kw) 2022-03-20T14:58:39.575 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-03-20T14:58:39.575 INFO:tasks.rook.operator.smithi007.stdout: osd_fsid, json.dumps(secrets)) 2022-03-20T14:58:39.575 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-03-20T14:58:39.575 INFO:tasks.rook.operator.smithi007.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-03-20T14:58:39.575 INFO:tasks.rook.operator.smithi007.stdout:RuntimeError: Unable to create a new OSD id 2022-03-20T14:58:39.576 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,294][ceph_volume.devices.raw.prepare][INFO ] will rollback OSD ID creation 2022-03-20T14:58:39.576 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,294][ceph_volume][ERROR ] exception caught by decorator 2022-03-20T14:58:39.576 INFO:tasks.rook.operator.smithi007.stdout:Traceback (most recent call last): 2022-03-20T14:58:39.576 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2022-03-20T14:58:39.576 INFO:tasks.rook.operator.smithi007.stdout: return f(*a, **kw) 2022-03-20T14:58:39.577 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2022-03-20T14:58:39.577 INFO:tasks.rook.operator.smithi007.stdout: terminal.dispatch(self.mapper, subcommand_args) 2022-03-20T14:58:39.577 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-20T14:58:39.577 INFO:tasks.rook.operator.smithi007.stdout: instance.main() 2022-03-20T14:58:39.578 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main 2022-03-20T14:58:39.578 INFO:tasks.rook.operator.smithi007.stdout: terminal.dispatch(self.mapper, self.argv) 2022-03-20T14:58:39.578 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-20T14:58:39.578 INFO:tasks.rook.operator.smithi007.stdout: instance.main() 2022-03-20T14:58:39.579 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main 2022-03-20T14:58:39.579 INFO:tasks.rook.operator.smithi007.stdout: self.safe_prepare(self.args) 2022-03-20T14:58:39.579 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-03-20T14:58:39.579 INFO:tasks.rook.operator.smithi007.stdout: self.prepare() 2022-03-20T14:58:39.579 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-03-20T14:58:39.580 INFO:tasks.rook.operator.smithi007.stdout: return func(*a, **kw) 2022-03-20T14:58:39.580 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-03-20T14:58:39.580 INFO:tasks.rook.operator.smithi007.stdout: osd_fsid, json.dumps(secrets)) 2022-03-20T14:58:39.580 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-03-20T14:58:39.580 INFO:tasks.rook.operator.smithi007.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-03-20T14:58:39.581 INFO:tasks.rook.operator.smithi007.stdout:RuntimeError: Unable to create a new OSD id 2022-03-20T14:58:39.581 INFO:tasks.rook.operator.smithi007.stdout:: exit status 1} 2022-03-20T14:58:39.581 INFO:tasks.rook.operator.smithi007.stdout:failed to provision OSD(s) on PVC smithi087-lv-2-data-04lndq. &{OSDs:[] Status:failed PvcBackedOSD:true Message:failed to configure devices: failed to initialize devices on PVC: failed to run ceph-volume. Running command: /usr/bin/ceph-authtool --gen-print-key 2022-03-20T14:58:39.581 INFO:tasks.rook.operator.smithi007.stdout:Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 198569b9-1e07-4d16-a236-026a4595652c 2022-03-20T14:58:39.582 INFO:tasks.rook.operator.smithi007.stdout: stderr: unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-20T14:58:39.582 INFO:tasks.rook.operator.smithi007.stdout: stderr: 2022-03-20T14:58:22.294+0000 7f8ea846d700 -1 failed for service _ceph-mon._tcp 2022-03-20T14:58:39.582 INFO:tasks.rook.operator.smithi007.stdout: stderr: 2022-03-20T14:58:22.294+0000 7f8ea846d700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-20T14:58:39.582 INFO:tasks.rook.operator.smithi007.stdout: stderr: [errno 2] RADOS object not found (error connecting to the cluster) 2022-03-20T14:58:39.582 INFO:tasks.rook.operator.smithi007.stdout:Traceback (most recent call last): 2022-03-20T14:58:39.583 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/sbin/ceph-volume", line 11, in 2022-03-20T14:58:39.583 INFO:tasks.rook.operator.smithi007.stdout: load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2022-03-20T14:58:39.583 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2022-03-20T14:58:39.583 INFO:tasks.rook.operator.smithi007.stdout: self.main(self.argv) 2022-03-20T14:58:39.583 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2022-03-20T14:58:39.584 INFO:tasks.rook.operator.smithi007.stdout: return f(*a, **kw) 2022-03-20T14:58:39.584 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2022-03-20T14:58:39.584 INFO:tasks.rook.operator.smithi007.stdout: terminal.dispatch(self.mapper, subcommand_args) 2022-03-20T14:58:39.584 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-20T14:58:39.585 INFO:tasks.rook.operator.smithi007.stdout: instance.main() 2022-03-20T14:58:39.585 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main 2022-03-20T14:58:39.585 INFO:tasks.rook.operator.smithi007.stdout: terminal.dispatch(self.mapper, self.argv) 2022-03-20T14:58:39.585 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-20T14:58:39.585 INFO:tasks.rook.operator.smithi007.stdout: instance.main() 2022-03-20T14:58:39.586 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main 2022-03-20T14:58:39.586 INFO:tasks.rook.operator.smithi007.stdout: self.safe_prepare(self.args) 2022-03-20T14:58:39.586 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-03-20T14:58:39.586 INFO:tasks.rook.operator.smithi007.stdout: self.prepare() 2022-03-20T14:58:39.586 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-03-20T14:58:39.587 INFO:tasks.rook.operator.smithi007.stdout: return func(*a, **kw) 2022-03-20T14:58:39.587 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-03-20T14:58:39.587 INFO:tasks.rook.operator.smithi007.stdout: osd_fsid, json.dumps(secrets)) 2022-03-20T14:58:39.587 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-03-20T14:58:39.588 INFO:tasks.rook.operator.smithi007.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-03-20T14:58:39.588 INFO:tasks.rook.operator.smithi007.stdout:RuntimeError: Unable to create a new OSD id. debug logs below: 2022-03-20T14:58:39.588 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,837][ceph_volume.main][INFO ] Running command: ceph-volume --log-path /var/log/ceph/smithi087-lv-2-data-04lndq raw prepare --bluestore --data /dev/mapper/vg_nvme-lv_2 2022-03-20T14:58:39.588 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,839][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk -plno KNAME,NAME,TYPE 2022-03-20T14:58:39.588 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,845][ceph_volume.process][INFO ] stdout /dev/loop0 /dev/loop0 loop 2022-03-20T14:58:39.589 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,845][ceph_volume.process][INFO ] stdout /dev/loop1 /dev/loop1 loop 2022-03-20T14:58:39.589 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,845][ceph_volume.process][INFO ] stdout /dev/loop2 /dev/loop2 loop 2022-03-20T14:58:39.589 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,845][ceph_volume.process][INFO ] stdout /dev/loop3 /dev/loop3 loop 2022-03-20T14:58:39.589 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,845][ceph_volume.process][INFO ] stdout /dev/sda /dev/sda disk 2022-03-20T14:58:39.589 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,845][ceph_volume.process][INFO ] stdout /dev/sda1 /dev/sda1 part 2022-03-20T14:58:39.590 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,845][ceph_volume.process][INFO ] stdout /dev/dm-0 /dev/mapper/vg_nvme-lv_1 lvm 2022-03-20T14:58:39.590 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,845][ceph_volume.process][INFO ] stdout /dev/dm-1 /dev/mapper/vg_nvme-lv_2 lvm 2022-03-20T14:58:39.590 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,845][ceph_volume.process][INFO ] stdout /dev/dm-2 /dev/mapper/vg_nvme-lv_3 lvm 2022-03-20T14:58:39.590 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,845][ceph_volume.process][INFO ] stdout /dev/dm-3 /dev/mapper/vg_nvme-lv_4 lvm 2022-03-20T14:58:39.591 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,845][ceph_volume.process][INFO ] stdout /dev/dm-4 /dev/mapper/vg_nvme-lv_5 lvm 2022-03-20T14:58:39.591 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,845][ceph_volume.process][INFO ] stdout /dev/nvme0n1 /dev/nvme0n1 disk 2022-03-20T14:58:39.591 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,845][ceph_volume.process][INFO ] stdout /dev/nvme1n1 /dev/nvme1n1 disk 2022-03-20T14:58:39.591 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,845][ceph_volume.process][INFO ] stdout /dev/nvme2n1 /dev/nvme2n1 disk 2022-03-20T14:58:39.591 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,846][ceph_volume.process][INFO ] stdout /dev/nvme3n1 /dev/nvme3n1 disk 2022-03-20T14:58:39.592 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,846][ceph_volume.process][INFO ] stdout /dev/nvme4n1 /dev/nvme4n1 disk 2022-03-20T14:58:39.592 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,856][ceph_volume.util.system][WARNING] Executable lvs not found on the host, will return lvs as-is 2022-03-20T14:58:39.592 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:21,856][ceph_volume.process][INFO ] Running command: lvs --noheadings --readonly --separator=";" -a --units=b --nosuffix -S lv_path=/dev/mapper/vg_nvme-lv_2 -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size 2022-03-20T14:58:39.592 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,157][ceph_volume.process][INFO ] Running command: /usr/bin/lsblk --nodeps -P -o NAME,KNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL /dev/mapper/vg_nvme-lv_2 2022-03-20T14:58:39.592 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,163][ceph_volume.process][INFO ] stdout NAME="vg_nvme-lv_2" KNAME="dm-1" MAJ:MIN="253:1" FSTYPE="" MOUNTPOINT="" LABEL="" UUID="" RO="0" RM="0" MODEL="" SIZE="89.4G" STATE="running" OWNER="root" GROUP="disk" MODE="brw-rw----" ALIGNMENT="0" PHY-SEC="512" LOG-SEC="512" ROTA="0" SCHED="" TYPE="lvm" DISC-ALN="0" DISC-GRAN="512B" DISC-MAX="2T" DISC-ZERO="0" PKNAME="" PARTLABEL="" 2022-03-20T14:58:39.593 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,163][ceph_volume.process][INFO ] Running command: /usr/sbin/blkid -c /dev/null -p /dev/mapper/vg_nvme-lv_2 2022-03-20T14:58:39.593 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,166][ceph_volume.util.disk][INFO ] opening device /dev/mapper/vg_nvme-lv_2 to check for BlueStore label 2022-03-20T14:58:39.593 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,167][ceph_volume.util.disk][INFO ] opening device /dev/mapper/vg_nvme-lv_2 to check for BlueStore label 2022-03-20T14:58:39.593 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,167][ceph_volume.process][INFO ] Running command: /usr/sbin/udevadm info --query=property /dev/mapper/vg_nvme-lv_2 2022-03-20T14:58:39.593 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,173][ceph_volume.process][INFO ] stdout DEVLINKS=/dev/disk/by-id/dm-uuid-LVM-UojCRVSNPvr1ajoeYEqVoya4dmuL8IkAYkwQoadiqgTuV3p0HiOt9Ek1foK2kJ2t /dev/vg_nvme/lv_2 /dev/mapper/vg_nvme-lv_2 /dev/disk/by-id/dm-name-vg_nvme-lv_2 2022-03-20T14:58:39.594 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,174][ceph_volume.process][INFO ] stdout DEVNAME=/dev/dm-1 2022-03-20T14:58:39.594 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,174][ceph_volume.process][INFO ] stdout DEVPATH=/devices/virtual/block/dm-1 2022-03-20T14:58:39.594 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,174][ceph_volume.process][INFO ] stdout DEVTYPE=disk 2022-03-20T14:58:39.594 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,174][ceph_volume.process][INFO ] stdout DM_LV_NAME=lv_2 2022-03-20T14:58:39.595 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,174][ceph_volume.process][INFO ] stdout DM_NAME=vg_nvme-lv_2 2022-03-20T14:58:39.595 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,174][ceph_volume.process][INFO ] stdout DM_SUSPENDED=0 2022-03-20T14:58:39.595 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,174][ceph_volume.process][INFO ] stdout DM_UDEV_DISABLE_LIBRARY_FALLBACK_FLAG=1 2022-03-20T14:58:39.595 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,174][ceph_volume.process][INFO ] stdout DM_UDEV_PRIMARY_SOURCE_FLAG=1 2022-03-20T14:58:39.595 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,174][ceph_volume.process][INFO ] stdout DM_UDEV_RULES=1 2022-03-20T14:58:39.596 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,174][ceph_volume.process][INFO ] stdout DM_UDEV_RULES_VSN=2 2022-03-20T14:58:39.596 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,174][ceph_volume.process][INFO ] stdout DM_UUID=LVM-UojCRVSNPvr1ajoeYEqVoya4dmuL8IkAYkwQoadiqgTuV3p0HiOt9Ek1foK2kJ2t 2022-03-20T14:58:39.596 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,174][ceph_volume.process][INFO ] stdout DM_VG_NAME=vg_nvme 2022-03-20T14:58:39.596 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,174][ceph_volume.process][INFO ] stdout MAJOR=253 2022-03-20T14:58:39.597 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,174][ceph_volume.process][INFO ] stdout MINOR=1 2022-03-20T14:58:39.597 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,174][ceph_volume.process][INFO ] stdout SUBSYSTEM=block 2022-03-20T14:58:39.597 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,175][ceph_volume.process][INFO ] stdout TAGS=:systemd: 2022-03-20T14:58:39.597 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,175][ceph_volume.process][INFO ] stdout USEC_INITIALIZED=219766073 2022-03-20T14:58:39.597 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,175][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-authtool --gen-print-key 2022-03-20T14:58:39.598 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,191][ceph_volume.process][INFO ] stdout AQAOQTdix/tYCxAAFwamna54d6Ua2fcs5vYbgA== 2022-03-20T14:58:39.598 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,192][ceph_volume.process][INFO ] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 198569b9-1e07-4d16-a236-026a4595652c 2022-03-20T14:58:39.598 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,299][ceph_volume.process][INFO ] stderr unable to get monitor info from DNS SRV with service name: ceph-mon 2022-03-20T14:58:39.598 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,299][ceph_volume.process][INFO ] stderr 2022-03-20T14:58:22.294+0000 7f8ea846d700 -1 failed for service _ceph-mon._tcp 2022-03-20T14:58:39.599 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,299][ceph_volume.process][INFO ] stderr 2022-03-20T14:58:22.294+0000 7f8ea846d700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-03-20T14:58:39.599 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,299][ceph_volume.process][INFO ] stderr [errno 2] RADOS object not found (error connecting to the cluster) 2022-03-20T14:58:39.599 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,299][ceph_volume.devices.raw.prepare][ERROR ] raw prepare was unable to complete 2022-03-20T14:58:39.599 INFO:tasks.rook.operator.smithi007.stdout:Traceback (most recent call last): 2022-03-20T14:58:39.599 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-03-20T14:58:39.600 INFO:tasks.rook.operator.smithi007.stdout: self.prepare() 2022-03-20T14:58:39.600 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-03-20T14:58:39.600 INFO:tasks.rook.operator.smithi007.stdout: return func(*a, **kw) 2022-03-20T14:58:39.600 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-03-20T14:58:39.600 INFO:tasks.rook.operator.smithi007.stdout: osd_fsid, json.dumps(secrets)) 2022-03-20T14:58:39.601 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-03-20T14:58:39.601 INFO:tasks.rook.operator.smithi007.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-03-20T14:58:39.601 INFO:tasks.rook.operator.smithi007.stdout:RuntimeError: Unable to create a new OSD id 2022-03-20T14:58:39.601 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,300][ceph_volume.devices.raw.prepare][INFO ] will rollback OSD ID creation 2022-03-20T14:58:39.602 INFO:tasks.rook.operator.smithi007.stdout:[2022-03-20 14:58:22,300][ceph_volume][ERROR ] exception caught by decorator 2022-03-20T14:58:39.602 INFO:tasks.rook.operator.smithi007.stdout:Traceback (most recent call last): 2022-03-20T14:58:39.602 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2022-03-20T14:58:39.602 INFO:tasks.rook.operator.smithi007.stdout: return f(*a, **kw) 2022-03-20T14:58:39.603 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2022-03-20T14:58:39.603 INFO:tasks.rook.operator.smithi007.stdout: terminal.dispatch(self.mapper, subcommand_args) 2022-03-20T14:58:39.603 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-20T14:58:39.603 INFO:tasks.rook.operator.smithi007.stdout: instance.main() 2022-03-20T14:58:39.603 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main 2022-03-20T14:58:39.604 INFO:tasks.rook.operator.smithi007.stdout: terminal.dispatch(self.mapper, self.argv) 2022-03-20T14:58:39.604 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-03-20T14:58:39.604 INFO:tasks.rook.operator.smithi007.stdout: instance.main() 2022-03-20T14:58:39.604 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main 2022-03-20T14:58:39.604 INFO:tasks.rook.operator.smithi007.stdout: self.safe_prepare(self.args) 2022-03-20T14:58:39.605 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2022-03-20T14:58:39.605 INFO:tasks.rook.operator.smithi007.stdout: self.prepare() 2022-03-20T14:58:39.605 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2022-03-20T14:58:39.605 INFO:tasks.rook.operator.smithi007.stdout: return func(*a, **kw) 2022-03-20T14:58:39.605 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2022-03-20T14:58:39.606 INFO:tasks.rook.operator.smithi007.stdout: osd_fsid, json.dumps(secrets)) 2022-03-20T14:58:39.606 INFO:tasks.rook.operator.smithi007.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2022-03-20T14:58:39.606 INFO:tasks.rook.operator.smithi007.stdout: raise RuntimeError('Unable to create a new OSD id') 2022-03-20T14:58:39.606 INFO:tasks.rook.operator.smithi007.stdout:RuntimeError: Unable to create a new OSD id 2022-03-20T14:58:39.607 INFO:tasks.rook.operator.smithi007.stdout:: exit status 1} 2022-03-20T14:58:39.607 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:39.549894 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-20T14:58:40.195 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:40.198715 I | op-mon: parsing mon endpoints: a=10.98.115.144:6789,b=10.109.19.50:6789,c=10.106.65.43:6789 2022-03-20T14:58:41.059 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:41.063508 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e... 2022-03-20T14:58:49.209 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd stat -f json 2022-03-20T14:58:49.939 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T14:58:49.956 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":15,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1647788308,"num_remapped_pgs":0} 2022-03-20T14:58:49.956 INFO:tasks.rook: have 0/12 OSDs 2022-03-20T14:58:53.116 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:53.121255 I | ceph-spec: detected ceph image version: "17.0.0-11187 quincy" 2022-03-20T14:58:53.117 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:53.121281 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-20T14:58:53.482 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:53.486233 I | op-mon: parsing mon endpoints: a=10.98.115.144:6789,b=10.109.19.50:6789,c=10.106.65.43:6789 2022-03-20T14:58:53.742 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:53.747270 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T14:58:53.745 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:53.747499 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T14:58:54.418 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:54.422513 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11187 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e" 2022-03-20T14:58:55.800 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:55.804231 I | op-mon: start running mons 2022-03-20T14:58:56.027 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:56.031505 I | op-mon: parsing mon endpoints: a=10.98.115.144:6789,b=10.109.19.50:6789,c=10.106.65.43:6789 2022-03-20T14:58:56.103 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:56.108019 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.98.115.144:6789","10.109.19.50:6789","10.106.65.43:6789"]}] data:a=10.98.115.144:6789,b=10.109.19.50:6789,c=10.106.65.43:6789 mapping:{"node":{"a":{"Name":"smithi087","Hostname":"smithi087","Address":"172.21.15.87"},"b":{"Name":"smithi057","Hostname":"smithi057","Address":"172.21.15.57"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:2] 2022-03-20T14:58:56.110 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:56.114754 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T14:58:56.113 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:56.115042 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T14:58:58.375 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:58.379253 I | op-mon: targeting the mon count 3 2022-03-20T14:58:58.383 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:58.387372 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T14:58:59.169 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:59.173058 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T14:58:59.169 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:59.173091 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T14:58:59.515 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:59.519204 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T14:58:59.515 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:58:59.519244 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T14:58:59.957 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd stat -f json 2022-03-20T14:59:00.350 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:00.353529 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T14:59:00.350 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:00.353567 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-20T14:59:01.216 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T14:59:01.239 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:01.241596 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-20T14:59:01.239 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:01.241635 I | op-config: deleting "log file" option from the mon configuration database 2022-03-20T14:59:01.241 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":22,"num_osds":12,"num_up_osds":4,"osd_up_since":1647788340,"num_in_osds":12,"osd_in_since":1647788308,"num_remapped_pgs":0} 2022-03-20T14:59:01.241 INFO:tasks.rook: have 4/12 OSDs 2022-03-20T14:59:01.683 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:01.686721 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-20T14:59:01.683 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:01.686745 I | op-mon: checking for basic quorum with existing mons 2022-03-20T14:59:02.609 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:02.609971 I | op-mon: mon "b" ip is 10.109.19.50 2022-03-20T14:59:04.241 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:04.244054 I | op-mon: mon "c" ip is 10.106.65.43 2022-03-20T14:59:05.773 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:05.776019 I | op-mon: mon "a" ip is 10.98.115.144 2022-03-20T14:59:06.531 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:05.948739 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.98.115.144:6789","10.109.19.50:6789","10.106.65.43:6789"]}] data:a=10.98.115.144:6789,b=10.109.19.50:6789,c=10.106.65.43:6789 mapping:{"node":{"a":{"Name":"smithi087","Hostname":"smithi087","Address":"172.21.15.87"},"b":{"Name":"smithi057","Hostname":"smithi057","Address":"172.21.15.57"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:2] 2022-03-20T14:59:06.532 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:06.000543 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T14:59:06.532 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:06.000808 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T14:59:06.532 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:06.242893 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-20T14:59:06.532 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:06.340347 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-20T14:59:06.533 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:06.340378 I | op-mon: waiting for mon quorum with [b c a] 2022-03-20T14:59:06.533 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:06.428926 I | op-mon: mons running: [b c a] 2022-03-20T14:59:06.815 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:06.818243 I | op-mon: Monitors in quorum: [a b c] 2022-03-20T14:59:07.170 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:07.172649 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-20T14:59:07.182 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:07.185202 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-20T14:59:07.182 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:07.185232 I | op-mon: waiting for mon quorum with [b c a] 2022-03-20T14:59:07.243 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:07.246307 I | op-mon: mons running: [b c a] 2022-03-20T14:59:07.658 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:07.661434 I | op-mon: Monitors in quorum: [a b c] 2022-03-20T14:59:07.731 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:07.733635 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-20T14:59:07.742 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:07.744870 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-20T14:59:07.742 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:07.744888 I | op-mon: waiting for mon quorum with [b c a] 2022-03-20T14:59:07.800 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:07.803284 I | op-mon: mons running: [b c a] 2022-03-20T14:59:08.211 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:08.213266 I | op-mon: Monitors in quorum: [a b c] 2022-03-20T14:59:08.211 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:08.213297 I | op-mon: mons created: 3 2022-03-20T14:59:08.622 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:08.625104 I | op-mon: waiting for mon quorum with [b c a] 2022-03-20T14:59:08.663 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:08.666126 I | op-mon: mons running: [b c a] 2022-03-20T14:59:09.058 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:09.060948 I | op-mon: Monitors in quorum: [a b c] 2022-03-20T14:59:09.059 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:09.061000 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-20T14:59:09.520 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:09.522782 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-20T14:59:09.958 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:09.961009 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-20T14:59:10.423 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:10.426068 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-20T14:59:10.901 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:10.903410 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-20T14:59:10.901 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:10.903434 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-20T14:59:11.243 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd stat -f json 2022-03-20T14:59:11.330 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:11.332540 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-20T14:59:11.331 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:11.332574 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-20T14:59:11.331 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:11.332583 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-20T14:59:11.823 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T14:59:11.843 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:11.842940 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-20T14:59:11.847 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":26,"num_osds":12,"num_up_osds":6,"osd_up_since":1647788341,"num_in_osds":12,"osd_in_since":1647788308,"num_remapped_pgs":0} 2022-03-20T14:59:11.848 INFO:tasks.rook: have 6/12 OSDs 2022-03-20T14:59:11.874 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:11.875880 I | op-mgr: start running mgr 2022-03-20T14:59:11.874 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:11.875925 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-20T14:59:12.374 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:12.376207 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-20T14:59:12.386 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:12.388052 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-20T14:59:12.389 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:12.391693 I | op-mgr: setting services to point to mgr "a" 2022-03-20T14:59:12.879 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:12.881136 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-20T14:59:12.879 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:12.881266 I | op-mgr: successful modules: balancer 2022-03-20T14:59:12.891 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:12.891919 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-20T14:59:12.891 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:12.892002 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-20T14:59:12.941 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:12.943469 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-20T14:59:13.083 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.085243 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi007-lv-1" 2022-03-20T14:59:13.084 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.085295 I | op-osd: OSD PVC "smithi007-lv-1-data-0fltgg" already exists 2022-03-20T14:59:13.084 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.085319 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi007-lv-2" 2022-03-20T14:59:13.084 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.085344 I | op-osd: OSD PVC "smithi007-lv-2-data-0z6lbn" already exists 2022-03-20T14:59:13.086 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.085365 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi007-lv-3" 2022-03-20T14:59:13.086 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.085379 I | op-osd: OSD PVC "smithi007-lv-3-data-0wb6kn" already exists 2022-03-20T14:59:13.087 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.085395 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi007-lv-4" 2022-03-20T14:59:13.087 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.085418 I | op-osd: OSD PVC "smithi007-lv-4-data-0snpfz" already exists 2022-03-20T14:59:13.087 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.085433 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi057-lv-1" 2022-03-20T14:59:13.087 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.085454 I | op-osd: OSD PVC "smithi057-lv-1-data-056qwk" already exists 2022-03-20T14:59:13.088 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.085469 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi057-lv-2" 2022-03-20T14:59:13.088 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.085489 I | op-osd: OSD PVC "smithi057-lv-2-data-0xdlsb" already exists 2022-03-20T14:59:13.088 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.085505 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi057-lv-3" 2022-03-20T14:59:13.088 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.085537 I | op-osd: OSD PVC "smithi057-lv-3-data-0vrztn" already exists 2022-03-20T14:59:13.088 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.085556 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi057-lv-4" 2022-03-20T14:59:13.089 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.085578 I | op-osd: OSD PVC "smithi057-lv-4-data-0zt8b4" already exists 2022-03-20T14:59:13.089 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.085602 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi087-lv-1" 2022-03-20T14:59:13.089 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.085625 I | op-osd: OSD PVC "smithi087-lv-1-data-0p7bb2" already exists 2022-03-20T14:59:13.089 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.085648 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi087-lv-2" 2022-03-20T14:59:13.090 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.085674 I | op-osd: OSD PVC "smithi087-lv-2-data-04lndq" already exists 2022-03-20T14:59:13.090 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.085708 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi087-lv-3" 2022-03-20T14:59:13.090 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.085736 I | op-osd: OSD PVC "smithi087-lv-3-data-0brh6f" already exists 2022-03-20T14:59:13.090 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.085767 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi087-lv-4" 2022-03-20T14:59:13.090 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.085825 I | op-osd: OSD PVC "smithi087-lv-4-data-0vssdn" already exists 2022-03-20T14:59:13.114 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.114487 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-1-data-0fltgg" 2022-03-20T14:59:13.114 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.114511 I | op-osd: skipping OSD prepare job creation for PVC "smithi007-lv-1-data-0fltgg" because OSD daemon using the PVC already exists 2022-03-20T14:59:13.114 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.114518 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-2-data-0z6lbn" 2022-03-20T14:59:13.115 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.114523 I | op-osd: skipping OSD prepare job creation for PVC "smithi007-lv-2-data-0z6lbn" because OSD daemon using the PVC already exists 2022-03-20T14:59:13.115 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.114528 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-3-data-0wb6kn" 2022-03-20T14:59:13.115 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.114533 I | op-osd: skipping OSD prepare job creation for PVC "smithi007-lv-3-data-0wb6kn" because OSD daemon using the PVC already exists 2022-03-20T14:59:13.115 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.114538 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-4-data-0snpfz" 2022-03-20T14:59:13.116 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.114543 I | op-osd: skipping OSD prepare job creation for PVC "smithi007-lv-4-data-0snpfz" because OSD daemon using the PVC already exists 2022-03-20T14:59:13.116 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.114548 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-1-data-056qwk" 2022-03-20T14:59:13.116 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.114558 I | op-osd: skipping OSD prepare job creation for PVC "smithi057-lv-1-data-056qwk" because OSD daemon using the PVC already exists 2022-03-20T14:59:13.116 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.114566 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-2-data-0xdlsb" 2022-03-20T14:59:13.117 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.114575 I | op-osd: skipping OSD prepare job creation for PVC "smithi057-lv-2-data-0xdlsb" because OSD daemon using the PVC already exists 2022-03-20T14:59:13.117 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.114583 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-3-data-0vrztn" 2022-03-20T14:59:13.117 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.114588 I | op-osd: skipping OSD prepare job creation for PVC "smithi057-lv-3-data-0vrztn" because OSD daemon using the PVC already exists 2022-03-20T14:59:13.117 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.114593 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-4-data-0zt8b4" 2022-03-20T14:59:13.118 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.114603 I | op-osd: skipping OSD prepare job creation for PVC "smithi057-lv-4-data-0zt8b4" because OSD daemon using the PVC already exists 2022-03-20T14:59:13.118 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.114611 I | op-osd: OSD will have its main bluestore block on "smithi087-lv-1-data-0p7bb2" 2022-03-20T14:59:13.118 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.114619 I | op-osd: skipping OSD prepare job creation for PVC "smithi087-lv-1-data-0p7bb2" because OSD daemon using the PVC already exists 2022-03-20T14:59:13.118 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.114627 I | op-osd: OSD will have its main bluestore block on "smithi087-lv-2-data-04lndq" 2022-03-20T14:59:13.521 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.523268 I | op-k8sutil: Removing previous job rook-ceph-osd-prepare-smithi087-lv-2-data-04lndq to start a new one 2022-03-20T14:59:13.561 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:13.563216 I | op-k8sutil: batch job rook-ceph-osd-prepare-smithi087-lv-2-data-04lndq still exists 2022-03-20T14:59:14.591 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:14.592765 I | op-mgr: successful modules: dashboard 2022-03-20T14:59:14.591 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:14.592795 I | op-mgr: successful modules: prometheus 2022-03-20T14:59:14.593 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:14.594623 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-20T14:59:16.565 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:16.567145 I | op-k8sutil: batch job rook-ceph-osd-prepare-smithi087-lv-2-data-04lndq deleted 2022-03-20T14:59:16.573 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:16.575036 I | op-osd: started OSD provisioning job for PVC "smithi087-lv-2-data-04lndq" 2022-03-20T14:59:16.573 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:16.575067 I | op-osd: OSD will have its main bluestore block on "smithi087-lv-3-data-0brh6f" 2022-03-20T14:59:16.586 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:16.587707 I | op-mgr: successful modules: orchestrator modules 2022-03-20T14:59:16.759 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:16.760607 I | op-k8sutil: Removing previous job rook-ceph-osd-prepare-smithi087-lv-3-data-0brh6f to start a new one 2022-03-20T14:59:16.877 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:16.878979 I | op-k8sutil: batch job rook-ceph-osd-prepare-smithi087-lv-3-data-0brh6f still exists 2022-03-20T14:59:19.986 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:19.987641 I | op-k8sutil: batch job rook-ceph-osd-prepare-smithi087-lv-3-data-0brh6f deleted 2022-03-20T14:59:20.002 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:20.003313 I | op-osd: started OSD provisioning job for PVC "smithi087-lv-3-data-0brh6f" 2022-03-20T14:59:20.002 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:20.003339 I | op-osd: OSD will have its main bluestore block on "smithi087-lv-4-data-0vssdn" 2022-03-20T14:59:20.003 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:20.003346 I | op-osd: skipping OSD prepare job creation for PVC "smithi087-lv-4-data-0vssdn" because OSD daemon using the PVC already exists 2022-03-20T14:59:20.003 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:20.003351 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-20T14:59:20.003 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:20.003356 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-20T14:59:20.076 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:20.076456 I | op-osd: OSD orchestration status for PVC smithi087-lv-2-data-04lndq is "starting" 2022-03-20T14:59:20.076 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:20.076496 I | op-osd: OSD orchestration status for PVC smithi087-lv-3-data-0brh6f is "starting" 2022-03-20T14:59:20.176 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:20.177956 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T14:59:20.188 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:20.189481 I | op-osd: updating OSD 0 on PVC "smithi057-lv-2-data-0xdlsb" 2022-03-20T14:59:20.188 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:20.189506 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-2-data-0xdlsb" 2022-03-20T14:59:20.366 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:20.367123 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T14:59:20.372 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:20.374151 I | op-osd: updating OSD 1 on PVC "smithi057-lv-4-data-0zt8b4" 2022-03-20T14:59:20.373 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:20.374188 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-4-data-0zt8b4" 2022-03-20T14:59:20.542 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:20.543617 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T14:59:20.549 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:20.550288 I | op-osd: updating OSD 2 on PVC "smithi057-lv-1-data-056qwk" 2022-03-20T14:59:20.549 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:20.550316 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-1-data-056qwk" 2022-03-20T14:59:20.719 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:20.720589 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T14:59:20.785 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:20.785994 I | op-osd: updating OSD 3 on PVC "smithi057-lv-3-data-0vrztn" 2022-03-20T14:59:20.785 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:20.786029 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-3-data-0vrztn" 2022-03-20T14:59:21.489 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:21.490792 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T14:59:21.584 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:21.585484 I | op-osd: updating OSD 4 on PVC "smithi087-lv-1-data-0p7bb2" 2022-03-20T14:59:21.584 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:21.585519 I | op-osd: OSD will have its main bluestore block on "smithi087-lv-1-data-0p7bb2" 2022-03-20T14:59:21.849 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd stat -f json 2022-03-20T14:59:22.293 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:22.293838 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T14:59:22.384 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:22.385415 I | op-osd: updating OSD 5 on PVC "smithi007-lv-4-data-0snpfz" 2022-03-20T14:59:22.385 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:22.385455 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-4-data-0snpfz" 2022-03-20T14:59:22.453 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T14:59:22.473 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":32,"num_osds":12,"num_up_osds":10,"osd_up_since":1647788354,"num_in_osds":12,"osd_in_since":1647788308,"num_remapped_pgs":0} 2022-03-20T14:59:22.473 INFO:tasks.rook: have 10/12 OSDs 2022-03-20T14:59:23.089 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:23.089919 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T14:59:23.185 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:23.185710 I | op-osd: updating OSD 6 on PVC "smithi087-lv-4-data-0vssdn" 2022-03-20T14:59:23.185 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:23.185749 I | op-osd: OSD will have its main bluestore block on "smithi087-lv-4-data-0vssdn" 2022-03-20T14:59:23.890 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:23.890636 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T14:59:23.984 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:23.984782 I | op-osd: updating OSD 7 on PVC "smithi007-lv-1-data-0fltgg" 2022-03-20T14:59:23.984 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:23.984815 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-1-data-0fltgg" 2022-03-20T14:59:24.689 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:24.690537 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T14:59:24.804 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:24.804918 I | op-osd: updating OSD 8 on PVC "smithi007-lv-3-data-0wb6kn" 2022-03-20T14:59:24.804 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:24.804953 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-3-data-0wb6kn" 2022-03-20T14:59:25.493 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:25.494395 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T14:59:25.583 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:25.584161 I | op-osd: updating OSD 9 on PVC "smithi007-lv-2-data-0z6lbn" 2022-03-20T14:59:25.584 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:25.584184 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-2-data-0z6lbn" 2022-03-20T14:59:26.846 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:26.846734 I | op-osd: OSD orchestration status for node smithi087-lv-2-data-04lndq is "orchestrating" 2022-03-20T14:59:26.871 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:26.872301 I | op-osd: OSD orchestration status for PVC smithi087-lv-2-data-04lndq is "orchestrating" 2022-03-20T14:59:28.172 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:28.172810 I | op-osd: OSD orchestration status for PVC smithi087-lv-2-data-04lndq is "completed" 2022-03-20T14:59:28.172 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:28.172850 I | op-osd: creating OSD 11 on PVC "smithi087-lv-2-data-04lndq" 2022-03-20T14:59:28.173 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:28.172865 I | op-osd: OSD will have its main bluestore block on "smithi087-lv-2-data-04lndq" 2022-03-20T14:59:29.837 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:29.836931 I | op-osd: OSD orchestration status for node smithi087-lv-3-data-0brh6f is "orchestrating" 2022-03-20T14:59:29.868 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:29.869513 I | op-osd: OSD orchestration status for PVC smithi087-lv-3-data-0brh6f is "orchestrating" 2022-03-20T14:59:31.307 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:31.307703 I | op-osd: OSD orchestration status for PVC smithi087-lv-3-data-0brh6f is "completed" 2022-03-20T14:59:31.308 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:31.307738 I | op-osd: creating OSD 10 on PVC "smithi087-lv-3-data-0brh6f" 2022-03-20T14:59:31.308 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:31.307758 I | op-osd: OSD will have its main bluestore block on "smithi087-lv-3-data-0brh6f" 2022-03-20T14:59:31.885 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:31.885218 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-20T14:59:31.885 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 14:59:31.885253 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-20T14:59:32.474 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd stat -f json 2022-03-20T14:59:32.993 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T14:59:33.023 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":32,"num_osds":12,"num_up_osds":10,"osd_up_since":1647788354,"num_in_osds":12,"osd_in_since":1647788308,"num_remapped_pgs":0} 2022-03-20T14:59:33.024 INFO:tasks.rook: have 10/12 OSDs 2022-03-20T14:59:43.026 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd stat -f json 2022-03-20T14:59:43.555 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T14:59:43.576 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":32,"num_osds":12,"num_up_osds":10,"osd_up_since":1647788354,"num_in_osds":12,"osd_in_since":1647788308,"num_remapped_pgs":0} 2022-03-20T14:59:43.577 INFO:tasks.rook: have 10/12 OSDs 2022-03-20T14:59:53.579 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd stat -f json 2022-03-20T14:59:54.110 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T14:59:54.131 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":35,"num_osds":12,"num_up_osds":11,"osd_up_since":1647788393,"num_in_osds":12,"osd_in_since":1647788308,"num_remapped_pgs":0} 2022-03-20T14:59:54.131 INFO:tasks.rook: have 11/12 OSDs 2022-03-20T15:00:04.132 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd stat -f json 2022-03-20T15:00:04.660 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:00:04.680 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":37,"num_osds":12,"num_up_osds":12,"osd_up_since":1647788394,"num_in_osds":12,"osd_in_since":1647788308,"num_remapped_pgs":0} 2022-03-20T15:00:04.681 INFO:tasks.rook:Distributing ceph config and client.admin keyring 2022-03-20T15:00:04.681 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- cat /etc/ceph/ceph.conf 2022-03-20T15:00:04.888 INFO:teuthology.orchestra.run.smithi007.stdout:[global] 2022-03-20T15:00:04.889 INFO:teuthology.orchestra.run.smithi007.stdout:mon_host = 10.98.115.144:6789,10.109.19.50:6789,10.106.65.43:6789 2022-03-20T15:00:04.889 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:00:04.889 INFO:teuthology.orchestra.run.smithi007.stdout:[client.admin] 2022-03-20T15:00:04.890 INFO:teuthology.orchestra.run.smithi007.stdout:keyring = /etc/ceph/keyring 2022-03-20T15:00:04.921 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- cat /etc/ceph/keyring 2022-03-20T15:00:05.108 INFO:teuthology.orchestra.run.smithi007.stdout:[client.admin] 2022-03-20T15:00:05.109 INFO:teuthology.orchestra.run.smithi007.stdout:key = AQAQQDdihrNLJRAAhjVkE4nTfjUEAPc0sdGfeA== 2022-03-20T15:00:05.113 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /etc/ceph 2022-03-20T15:00:05.129 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /etc/ceph 2022-03-20T15:00:05.143 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /etc/ceph 2022-03-20T15:00:05.159 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-20T15:00:05.159 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.conf 2022-03-20T15:00:05.191 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-20T15:00:05.192 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/keyring 2022-03-20T15:00:05.208 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2022-03-20T15:00:05.209 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/ceph/ceph.conf 2022-03-20T15:00:05.224 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2022-03-20T15:00:05.225 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/ceph/keyring 2022-03-20T15:00:05.239 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2022-03-20T15:00:05.239 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/ceph/ceph.conf 2022-03-20T15:00:05.249 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2022-03-20T15:00:05.250 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/ceph/keyring 2022-03-20T15:00:05.261 INFO:tasks.rook:Setting up client nodes... 2022-03-20T15:00:05.262 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph auth get-or-create client.a mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-20T15:00:05.991 INFO:teuthology.orchestra.run.smithi007.stdout:[client.a] 2022-03-20T15:00:05.992 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQB1QTdia8TZNBAAEUiSZ3HDghAgN2ehmUuaUw== 2022-03-20T15:00:06.025 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-20T15:00:06.025 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.client.a.keyring 2022-03-20T15:00:06.025 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/ceph.client.a.keyring 2022-03-20T15:00:06.052 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph auth get-or-create client.b mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-20T15:00:06.768 INFO:teuthology.orchestra.run.smithi007.stdout:[client.b] 2022-03-20T15:00:06.769 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQB2QTdiyM/sJxAAJI4E/0ekJcyKIKW1LPHxSg== 2022-03-20T15:00:06.795 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2022-03-20T15:00:06.795 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/ceph/ceph.client.b.keyring 2022-03-20T15:00:06.796 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0644 /etc/ceph/ceph.client.b.keyring 2022-03-20T15:00:06.819 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph auth get-or-create client.c mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-20T15:00:07.553 INFO:teuthology.orchestra.run.smithi007.stdout:[client.c] 2022-03-20T15:00:07.553 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQB3QTdiK1c8GhAA6syW17sYfH8ZPbpRb9ttIw== 2022-03-20T15:00:07.583 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2022-03-20T15:00:07.584 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/ceph/ceph.client.c.keyring 2022-03-20T15:00:07.585 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0644 /etc/ceph/ceph.client.c.keyring 2022-03-20T15:00:07.612 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd dump --format=json 2022-03-20T15:00:08.114 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:00:08.247 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":37,"fsid":"ea418f6e-0cdf-45ac-b31b-73dc6739a8f8","created":"2022-03-20T14:54:22.611201+0000","modified":"2022-03-20T14:59:56.032185+0000","last_up_change":"2022-03-20T14:59:54.770421+0000","last_in_change":"2022-03-20T14:58:28.706904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":13,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-20T14:59:00.068675+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"fda408e5-af6e-47dc-90e0-444ada48a882","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.16:6800","nonce":193325540},{"type":"v1","addr":"10.248.49.16:6801","nonce":193325540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.16:6802","nonce":193325540},{"type":"v1","addr":"10.248.49.16:6803","nonce":193325540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.16:6806","nonce":193325540},{"type":"v1","addr":"10.248.49.16:6807","nonce":193325540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.16:6804","nonce":193325540},{"type":"v1","addr":"10.248.49.16:6805","nonce":193325540}]},"public_addr":"10.248.49.16:6801/193325540","cluster_addr":"10.248.49.16:6803/193325540","heartbeat_back_addr":"10.248.49.16:6807/193325540","heartbeat_front_addr":"10.248.49.16:6805/193325540","state":["exists","up"]},{"osd":1,"uuid":"1496f197-0d2f-400d-8e24-a6d6fe09a57a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.17:6800","nonce":3970919970},{"type":"v1","addr":"10.248.49.17:6801","nonce":3970919970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.17:6802","nonce":3970919970},{"type":"v1","addr":"10.248.49.17:6803","nonce":3970919970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.17:6806","nonce":3970919970},{"type":"v1","addr":"10.248.49.17:6807","nonce":3970919970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.17:6804","nonce":3970919970},{"type":"v1","addr":"10.248.49.17:6805","nonce":3970919970}]},"public_addr":"10.248.49.17:6801/3970919970","cluster_addr":"10.248.49.17:6803/3970919970","heartbeat_back_addr":"10.248.49.17:6807/3970919970","heartbeat_front_addr":"10.248.49.17:6805/3970919970","state":["exists","up"]},{"osd":2,"uuid":"1b243998-c7f1-4f45-b516-71d10628957f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.18:6800","nonce":4134901328},{"type":"v1","addr":"10.248.49.18:6801","nonce":4134901328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.18:6802","nonce":4134901328},{"type":"v1","addr":"10.248.49.18:6803","nonce":4134901328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.18:6806","nonce":4134901328},{"type":"v1","addr":"10.248.49.18:6807","nonce":4134901328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.18:6804","nonce":4134901328},{"type":"v1","addr":"10.248.49.18:6805","nonce":4134901328}]},"public_addr":"10.248.49.18:6801/4134901328","cluster_addr":"10.248.49.18:6803/4134901328","heartbeat_back_addr":"10.248.49.18:6807/4134901328","heartbeat_front_addr":"10.248.49.18:6805/4134901328","state":["exists","up"]},{"osd":3,"uuid":"9e36c9ef-ce43-4098-85a9-e350ca790ac5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.19:6800","nonce":1359880554},{"type":"v1","addr":"10.248.49.19:6801","nonce":1359880554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.19:6802","nonce":1359880554},{"type":"v1","addr":"10.248.49.19:6803","nonce":1359880554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.19:6806","nonce":1359880554},{"type":"v1","addr":"10.248.49.19:6807","nonce":1359880554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.19:6804","nonce":1359880554},{"type":"v1","addr":"10.248.49.19:6805","nonce":1359880554}]},"public_addr":"10.248.49.19:6801/1359880554","cluster_addr":"10.248.49.19:6803/1359880554","heartbeat_back_addr":"10.248.49.19:6807/1359880554","heartbeat_front_addr":"10.248.49.19:6805/1359880554","state":["exists","up"]},{"osd":4,"uuid":"207c10c5-2349-4d30-bf01-1ebd5c4bb57d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.13:6800","nonce":2445326158},{"type":"v1","addr":"10.248.50.13:6801","nonce":2445326158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.13:6802","nonce":2445326158},{"type":"v1","addr":"10.248.50.13:6803","nonce":2445326158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.13:6806","nonce":2445326158},{"type":"v1","addr":"10.248.50.13:6807","nonce":2445326158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.13:6804","nonce":2445326158},{"type":"v1","addr":"10.248.50.13:6805","nonce":2445326158}]},"public_addr":"10.248.50.13:6801/2445326158","cluster_addr":"10.248.50.13:6803/2445326158","heartbeat_back_addr":"10.248.50.13:6807/2445326158","heartbeat_front_addr":"10.248.50.13:6805/2445326158","state":["exists","up"]},{"osd":5,"uuid":"c0bded03-f49f-4e53-bd04-8d280177153c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.11:6800","nonce":2167751379},{"type":"v1","addr":"10.248.48.11:6801","nonce":2167751379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.11:6802","nonce":2167751379},{"type":"v1","addr":"10.248.48.11:6803","nonce":2167751379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.11:6806","nonce":2167751379},{"type":"v1","addr":"10.248.48.11:6807","nonce":2167751379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.11:6804","nonce":2167751379},{"type":"v1","addr":"10.248.48.11:6805","nonce":2167751379}]},"public_addr":"10.248.48.11:6801/2167751379","cluster_addr":"10.248.48.11:6803/2167751379","heartbeat_back_addr":"10.248.48.11:6807/2167751379","heartbeat_front_addr":"10.248.48.11:6805/2167751379","state":["exists","up"]},{"osd":6,"uuid":"b9f130db-343c-49ea-8a12-508c0c51d79b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.14:6800","nonce":3718528490},{"type":"v1","addr":"10.248.50.14:6801","nonce":3718528490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.14:6802","nonce":3718528490},{"type":"v1","addr":"10.248.50.14:6803","nonce":3718528490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.14:6806","nonce":3718528490},{"type":"v1","addr":"10.248.50.14:6807","nonce":3718528490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.14:6804","nonce":3718528490},{"type":"v1","addr":"10.248.50.14:6805","nonce":3718528490}]},"public_addr":"10.248.50.14:6801/3718528490","cluster_addr":"10.248.50.14:6803/3718528490","heartbeat_back_addr":"10.248.50.14:6807/3718528490","heartbeat_front_addr":"10.248.50.14:6805/3718528490","state":["exists","up"]},{"osd":7,"uuid":"514523ef-1d10-4e0b-a9c5-a6b789f988ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.13:6800","nonce":561302417},{"type":"v1","addr":"10.248.48.13:6801","nonce":561302417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.13:6802","nonce":561302417},{"type":"v1","addr":"10.248.48.13:6803","nonce":561302417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.13:6806","nonce":561302417},{"type":"v1","addr":"10.248.48.13:6807","nonce":561302417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.13:6804","nonce":561302417},{"type":"v1","addr":"10.248.48.13:6805","nonce":561302417}]},"public_addr":"10.248.48.13:6801/561302417","cluster_addr":"10.248.48.13:6803/561302417","heartbeat_back_addr":"10.248.48.13:6807/561302417","heartbeat_front_addr":"10.248.48.13:6805/561302417","state":["exists","up"]},{"osd":8,"uuid":"ba3091ce-f7a7-43f1-907e-f60ca7952110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.14:6800","nonce":961922660},{"type":"v1","addr":"10.248.48.14:6801","nonce":961922660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.14:6802","nonce":961922660},{"type":"v1","addr":"10.248.48.14:6803","nonce":961922660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.14:6806","nonce":961922660},{"type":"v1","addr":"10.248.48.14:6807","nonce":961922660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.14:6804","nonce":961922660},{"type":"v1","addr":"10.248.48.14:6805","nonce":961922660}]},"public_addr":"10.248.48.14:6801/961922660","cluster_addr":"10.248.48.14:6803/961922660","heartbeat_back_addr":"10.248.48.14:6807/961922660","heartbeat_front_addr":"10.248.48.14:6805/961922660","state":["exists","up"]},{"osd":9,"uuid":"6a1dd956-9931-411c-ad1d-4a83e702e703","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.15:6800","nonce":3919597058},{"type":"v1","addr":"10.248.48.15:6801","nonce":3919597058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.15:6802","nonce":3919597058},{"type":"v1","addr":"10.248.48.15:6803","nonce":3919597058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.15:6806","nonce":3919597058},{"type":"v1","addr":"10.248.48.15:6807","nonce":3919597058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.15:6804","nonce":3919597058},{"type":"v1","addr":"10.248.48.15:6805","nonce":3919597058}]},"public_addr":"10.248.48.15:6801/3919597058","cluster_addr":"10.248.48.15:6803/3919597058","heartbeat_back_addr":"10.248.48.15:6807/3919597058","heartbeat_front_addr":"10.248.48.15:6805/3919597058","state":["exists","up"]},{"osd":10,"uuid":"f06ae5c1-223a-424e-be38-2d6dd7de07f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.18:6800","nonce":1792565641},{"type":"v1","addr":"10.248.50.18:6801","nonce":1792565641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.18:6802","nonce":1792565641},{"type":"v1","addr":"10.248.50.18:6803","nonce":1792565641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.18:6806","nonce":1792565641},{"type":"v1","addr":"10.248.50.18:6807","nonce":1792565641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.18:6804","nonce":1792565641},{"type":"v1","addr":"10.248.50.18:6805","nonce":1792565641}]},"public_addr":"10.248.50.18:6801/1792565641","cluster_addr":"10.248.50.18:6803/1792565641","heartbeat_back_addr":"10.248.50.18:6807/1792565641","heartbeat_front_addr":"10.248.50.18:6805/1792565641","state":["exists","up"]},{"osd":11,"uuid":"e208cea4-d52f-4b0f-baad-da47c6844bf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.17:6800","nonce":2349665581},{"type":"v1","addr":"10.248.50.17:6801","nonce":2349665581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.17:6802","nonce":2349665581},{"type":"v1","addr":"10.248.50.17:6803","nonce":2349665581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.17:6806","nonce":2349665581},{"type":"v1","addr":"10.248.50.17:6807","nonce":2349665581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.17:6804","nonce":2349665581},{"type":"v1","addr":"10.248.50.17:6805","nonce":2349665581}]},"public_addr":"10.248.50.17:6801/2349665581","cluster_addr":"10.248.50.17:6803/2349665581","heartbeat_back_addr":"10.248.50.17:6807/2349665581","heartbeat_front_addr":"10.248.50.17:6805/2349665581","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:58:58.329848+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:58:56.055790+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:58:58.232279+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:58:55.949487+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:58:47.433907+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:59:10.833121+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:58:56.802957+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:59:10.930410+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:59:10.980307+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:59:12.217528+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:59:53.384240+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:59:51.964883+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.248.48.5:0/1750087674":"2022-03-21T14:57:16.309841+0000","10.248.48.5:0/3041300154":"2022-03-21T14:57:16.309841+0000","10.248.48.5:0/1222231597":"2022-03-21T14:57:16.309841+0000","10.248.48.5:6801/4177463852":"2022-03-21T14:57:16.309841+0000","10.248.48.5:6800/4177463852":"2022-03-21T14:57:16.309841+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-20T15:00:08.249 INFO:tasks.rook.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-03-20T14:59:00.068675+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'on', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-03-20T15:00:08.249 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd pool get .mgr pg_num 2022-03-20T15:00:09.255 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2022-03-20T15:00:09.288 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-03-20T15:00:09.289 INFO:tasks.rook.ceph_manager.ceph:waiting for mgr available 2022-03-20T15:00:09.289 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph mgr dump --format=json 2022-03-20T15:00:09.860 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:00:09.879 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":13,"active_gid":24152,"active_name":"a","active_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.5:6800","nonce":713633229},{"type":"v1","addr":"10.248.48.5:6801","nonce":713633229}]},"active_addr":"10.248.48.5:6801/713633229","active_change":"2022-03-20T14:57:16.332561+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","prometheus","restful","rook"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"prometheus":"http://10.248.48.5:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":2,"active_clients":[{"addrvec":[{"type":"v2","addr":"10.248.48.5:0","nonce":1048587915}]},{"addrvec":[{"type":"v2","addr":"10.248.48.5:0","nonce":1343924187}]},{"addrvec":[{"type":"v2","addr":"10.248.48.5:0","nonce":1346584845}]},{"addrvec":[{"type":"v2","addr":"10.248.48.5:0","nonce":3228741694}]}]}} 2022-03-20T15:00:09.882 INFO:tasks.rook.ceph_manager.ceph:mgr available! 2022-03-20T15:00:09.883 INFO:tasks.rook.ceph_manager.ceph:waiting for all up 2022-03-20T15:00:09.883 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd dump --format=json 2022-03-20T15:00:10.465 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:00:10.465 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":37,"fsid":"ea418f6e-0cdf-45ac-b31b-73dc6739a8f8","created":"2022-03-20T14:54:22.611201+0000","modified":"2022-03-20T14:59:56.032185+0000","last_up_change":"2022-03-20T14:59:54.770421+0000","last_in_change":"2022-03-20T14:58:28.706904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":13,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-20T14:59:00.068675+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"fda408e5-af6e-47dc-90e0-444ada48a882","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.16:6800","nonce":193325540},{"type":"v1","addr":"10.248.49.16:6801","nonce":193325540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.16:6802","nonce":193325540},{"type":"v1","addr":"10.248.49.16:6803","nonce":193325540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.16:6806","nonce":193325540},{"type":"v1","addr":"10.248.49.16:6807","nonce":193325540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.16:6804","nonce":193325540},{"type":"v1","addr":"10.248.49.16:6805","nonce":193325540}]},"public_addr":"10.248.49.16:6801/193325540","cluster_addr":"10.248.49.16:6803/193325540","heartbeat_back_addr":"10.248.49.16:6807/193325540","heartbeat_front_addr":"10.248.49.16:6805/193325540","state":["exists","up"]},{"osd":1,"uuid":"1496f197-0d2f-400d-8e24-a6d6fe09a57a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.17:6800","nonce":3970919970},{"type":"v1","addr":"10.248.49.17:6801","nonce":3970919970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.17:6802","nonce":3970919970},{"type":"v1","addr":"10.248.49.17:6803","nonce":3970919970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.17:6806","nonce":3970919970},{"type":"v1","addr":"10.248.49.17:6807","nonce":3970919970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.17:6804","nonce":3970919970},{"type":"v1","addr":"10.248.49.17:6805","nonce":3970919970}]},"public_addr":"10.248.49.17:6801/3970919970","cluster_addr":"10.248.49.17:6803/3970919970","heartbeat_back_addr":"10.248.49.17:6807/3970919970","heartbeat_front_addr":"10.248.49.17:6805/3970919970","state":["exists","up"]},{"osd":2,"uuid":"1b243998-c7f1-4f45-b516-71d10628957f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.18:6800","nonce":4134901328},{"type":"v1","addr":"10.248.49.18:6801","nonce":4134901328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.18:6802","nonce":4134901328},{"type":"v1","addr":"10.248.49.18:6803","nonce":4134901328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.18:6806","nonce":4134901328},{"type":"v1","addr":"10.248.49.18:6807","nonce":4134901328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.18:6804","nonce":4134901328},{"type":"v1","addr":"10.248.49.18:6805","nonce":4134901328}]},"public_addr":"10.248.49.18:6801/4134901328","cluster_addr":"10.248.49.18:6803/4134901328","heartbeat_back_addr":"10.248.49.18:6807/4134901328","heartbeat_front_addr":"10.248.49.18:6805/4134901328","state":["exists","up"]},{"osd":3,"uuid":"9e36c9ef-ce43-4098-85a9-e350ca790ac5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.19:6800","nonce":1359880554},{"type":"v1","addr":"10.248.49.19:6801","nonce":1359880554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.19:6802","nonce":1359880554},{"type":"v1","addr":"10.248.49.19:6803","nonce":1359880554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.19:6806","nonce":1359880554},{"type":"v1","addr":"10.248.49.19:6807","nonce":1359880554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.19:6804","nonce":1359880554},{"type":"v1","addr":"10.248.49.19:6805","nonce":1359880554}]},"public_addr":"10.248.49.19:6801/1359880554","cluster_addr":"10.248.49.19:6803/1359880554","heartbeat_back_addr":"10.248.49.19:6807/1359880554","heartbeat_front_addr":"10.248.49.19:6805/1359880554","state":["exists","up"]},{"osd":4,"uuid":"207c10c5-2349-4d30-bf01-1ebd5c4bb57d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.13:6800","nonce":2445326158},{"type":"v1","addr":"10.248.50.13:6801","nonce":2445326158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.13:6802","nonce":2445326158},{"type":"v1","addr":"10.248.50.13:6803","nonce":2445326158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.13:6806","nonce":2445326158},{"type":"v1","addr":"10.248.50.13:6807","nonce":2445326158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.13:6804","nonce":2445326158},{"type":"v1","addr":"10.248.50.13:6805","nonce":2445326158}]},"public_addr":"10.248.50.13:6801/2445326158","cluster_addr":"10.248.50.13:6803/2445326158","heartbeat_back_addr":"10.248.50.13:6807/2445326158","heartbeat_front_addr":"10.248.50.13:6805/2445326158","state":["exists","up"]},{"osd":5,"uuid":"c0bded03-f49f-4e53-bd04-8d280177153c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.11:6800","nonce":2167751379},{"type":"v1","addr":"10.248.48.11:6801","nonce":2167751379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.11:6802","nonce":2167751379},{"type":"v1","addr":"10.248.48.11:6803","nonce":2167751379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.11:6806","nonce":2167751379},{"type":"v1","addr":"10.248.48.11:6807","nonce":2167751379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.11:6804","nonce":2167751379},{"type":"v1","addr":"10.248.48.11:6805","nonce":2167751379}]},"public_addr":"10.248.48.11:6801/2167751379","cluster_addr":"10.248.48.11:6803/2167751379","heartbeat_back_addr":"10.248.48.11:6807/2167751379","heartbeat_front_addr":"10.248.48.11:6805/2167751379","state":["exists","up"]},{"osd":6,"uuid":"b9f130db-343c-49ea-8a12-508c0c51d79b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.14:6800","nonce":3718528490},{"type":"v1","addr":"10.248.50.14:6801","nonce":3718528490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.14:6802","nonce":3718528490},{"type":"v1","addr":"10.248.50.14:6803","nonce":3718528490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.14:6806","nonce":3718528490},{"type":"v1","addr":"10.248.50.14:6807","nonce":3718528490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.14:6804","nonce":3718528490},{"type":"v1","addr":"10.248.50.14:6805","nonce":3718528490}]},"public_addr":"10.248.50.14:6801/3718528490","cluster_addr":"10.248.50.14:6803/3718528490","heartbeat_back_addr":"10.248.50.14:6807/3718528490","heartbeat_front_addr":"10.248.50.14:6805/3718528490","state":["exists","up"]},{"osd":7,"uuid":"514523ef-1d10-4e0b-a9c5-a6b789f988ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.13:6800","nonce":561302417},{"type":"v1","addr":"10.248.48.13:6801","nonce":561302417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.13:6802","nonce":561302417},{"type":"v1","addr":"10.248.48.13:6803","nonce":561302417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.13:6806","nonce":561302417},{"type":"v1","addr":"10.248.48.13:6807","nonce":561302417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.13:6804","nonce":561302417},{"type":"v1","addr":"10.248.48.13:6805","nonce":561302417}]},"public_addr":"10.248.48.13:6801/561302417","cluster_addr":"10.248.48.13:6803/561302417","heartbeat_back_addr":"10.248.48.13:6807/561302417","heartbeat_front_addr":"10.248.48.13:6805/561302417","state":["exists","up"]},{"osd":8,"uuid":"ba3091ce-f7a7-43f1-907e-f60ca7952110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.14:6800","nonce":961922660},{"type":"v1","addr":"10.248.48.14:6801","nonce":961922660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.14:6802","nonce":961922660},{"type":"v1","addr":"10.248.48.14:6803","nonce":961922660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.14:6806","nonce":961922660},{"type":"v1","addr":"10.248.48.14:6807","nonce":961922660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.14:6804","nonce":961922660},{"type":"v1","addr":"10.248.48.14:6805","nonce":961922660}]},"public_addr":"10.248.48.14:6801/961922660","cluster_addr":"10.248.48.14:6803/961922660","heartbeat_back_addr":"10.248.48.14:6807/961922660","heartbeat_front_addr":"10.248.48.14:6805/961922660","state":["exists","up"]},{"osd":9,"uuid":"6a1dd956-9931-411c-ad1d-4a83e702e703","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.15:6800","nonce":3919597058},{"type":"v1","addr":"10.248.48.15:6801","nonce":3919597058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.15:6802","nonce":3919597058},{"type":"v1","addr":"10.248.48.15:6803","nonce":3919597058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.15:6806","nonce":3919597058},{"type":"v1","addr":"10.248.48.15:6807","nonce":3919597058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.15:6804","nonce":3919597058},{"type":"v1","addr":"10.248.48.15:6805","nonce":3919597058}]},"public_addr":"10.248.48.15:6801/3919597058","cluster_addr":"10.248.48.15:6803/3919597058","heartbeat_back_addr":"10.248.48.15:6807/3919597058","heartbeat_front_addr":"10.248.48.15:6805/3919597058","state":["exists","up"]},{"osd":10,"uuid":"f06ae5c1-223a-424e-be38-2d6dd7de07f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.18:6800","nonce":1792565641},{"type":"v1","addr":"10.248.50.18:6801","nonce":1792565641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.18:6802","nonce":1792565641},{"type":"v1","addr":"10.248.50.18:6803","nonce":1792565641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.18:6806","nonce":1792565641},{"type":"v1","addr":"10.248.50.18:6807","nonce":1792565641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.18:6804","nonce":1792565641},{"type":"v1","addr":"10.248.50.18:6805","nonce":1792565641}]},"public_addr":"10.248.50.18:6801/1792565641","cluster_addr":"10.248.50.18:6803/1792565641","heartbeat_back_addr":"10.248.50.18:6807/1792565641","heartbeat_front_addr":"10.248.50.18:6805/1792565641","state":["exists","up"]},{"osd":11,"uuid":"e208cea4-d52f-4b0f-baad-da47c6844bf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.17:6800","nonce":2349665581},{"type":"v1","addr":"10.248.50.17:6801","nonce":2349665581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.17:6802","nonce":2349665581},{"type":"v1","addr":"10.248.50.17:6803","nonce":2349665581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.17:6806","nonce":2349665581},{"type":"v1","addr":"10.248.50.17:6807","nonce":2349665581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.17:6804","nonce":2349665581},{"type":"v1","addr":"10.248.50.17:6805","nonce":2349665581}]},"public_addr":"10.248.50.17:6801/2349665581","cluster_addr":"10.248.50.17:6803/2349665581","heartbeat_back_addr":"10.248.50.17:6807/2349665581","heartbeat_front_addr":"10.248.50.17:6805/2349665581","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:58:58.329848+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:58:56.055790+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:58:58.232279+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:58:55.949487+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:58:47.433907+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:59:10.833121+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:58:56.802957+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:59:10.930410+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:59:10.980307+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:59:12.217528+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:59:53.384240+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:59:51.964883+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.248.48.5:0/1750087674":"2022-03-21T14:57:16.309841+0000","10.248.48.5:0/3041300154":"2022-03-21T14:57:16.309841+0000","10.248.48.5:0/1222231597":"2022-03-21T14:57:16.309841+0000","10.248.48.5:6801/4177463852":"2022-03-21T14:57:16.309841+0000","10.248.48.5:6800/4177463852":"2022-03-21T14:57:16.309841+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-20T15:00:10.488 INFO:tasks.rook.ceph_manager.ceph:all up! 2022-03-20T15:00:10.489 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd dump --format=json 2022-03-20T15:00:11.004 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:00:11.005 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":37,"fsid":"ea418f6e-0cdf-45ac-b31b-73dc6739a8f8","created":"2022-03-20T14:54:22.611201+0000","modified":"2022-03-20T14:59:56.032185+0000","last_up_change":"2022-03-20T14:59:54.770421+0000","last_in_change":"2022-03-20T14:58:28.706904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":13,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-20T14:59:00.068675+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"fda408e5-af6e-47dc-90e0-444ada48a882","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.16:6800","nonce":193325540},{"type":"v1","addr":"10.248.49.16:6801","nonce":193325540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.16:6802","nonce":193325540},{"type":"v1","addr":"10.248.49.16:6803","nonce":193325540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.16:6806","nonce":193325540},{"type":"v1","addr":"10.248.49.16:6807","nonce":193325540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.16:6804","nonce":193325540},{"type":"v1","addr":"10.248.49.16:6805","nonce":193325540}]},"public_addr":"10.248.49.16:6801/193325540","cluster_addr":"10.248.49.16:6803/193325540","heartbeat_back_addr":"10.248.49.16:6807/193325540","heartbeat_front_addr":"10.248.49.16:6805/193325540","state":["exists","up"]},{"osd":1,"uuid":"1496f197-0d2f-400d-8e24-a6d6fe09a57a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.17:6800","nonce":3970919970},{"type":"v1","addr":"10.248.49.17:6801","nonce":3970919970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.17:6802","nonce":3970919970},{"type":"v1","addr":"10.248.49.17:6803","nonce":3970919970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.17:6806","nonce":3970919970},{"type":"v1","addr":"10.248.49.17:6807","nonce":3970919970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.17:6804","nonce":3970919970},{"type":"v1","addr":"10.248.49.17:6805","nonce":3970919970}]},"public_addr":"10.248.49.17:6801/3970919970","cluster_addr":"10.248.49.17:6803/3970919970","heartbeat_back_addr":"10.248.49.17:6807/3970919970","heartbeat_front_addr":"10.248.49.17:6805/3970919970","state":["exists","up"]},{"osd":2,"uuid":"1b243998-c7f1-4f45-b516-71d10628957f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.18:6800","nonce":4134901328},{"type":"v1","addr":"10.248.49.18:6801","nonce":4134901328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.18:6802","nonce":4134901328},{"type":"v1","addr":"10.248.49.18:6803","nonce":4134901328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.18:6806","nonce":4134901328},{"type":"v1","addr":"10.248.49.18:6807","nonce":4134901328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.18:6804","nonce":4134901328},{"type":"v1","addr":"10.248.49.18:6805","nonce":4134901328}]},"public_addr":"10.248.49.18:6801/4134901328","cluster_addr":"10.248.49.18:6803/4134901328","heartbeat_back_addr":"10.248.49.18:6807/4134901328","heartbeat_front_addr":"10.248.49.18:6805/4134901328","state":["exists","up"]},{"osd":3,"uuid":"9e36c9ef-ce43-4098-85a9-e350ca790ac5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.19:6800","nonce":1359880554},{"type":"v1","addr":"10.248.49.19:6801","nonce":1359880554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.19:6802","nonce":1359880554},{"type":"v1","addr":"10.248.49.19:6803","nonce":1359880554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.19:6806","nonce":1359880554},{"type":"v1","addr":"10.248.49.19:6807","nonce":1359880554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.19:6804","nonce":1359880554},{"type":"v1","addr":"10.248.49.19:6805","nonce":1359880554}]},"public_addr":"10.248.49.19:6801/1359880554","cluster_addr":"10.248.49.19:6803/1359880554","heartbeat_back_addr":"10.248.49.19:6807/1359880554","heartbeat_front_addr":"10.248.49.19:6805/1359880554","state":["exists","up"]},{"osd":4,"uuid":"207c10c5-2349-4d30-bf01-1ebd5c4bb57d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.13:6800","nonce":2445326158},{"type":"v1","addr":"10.248.50.13:6801","nonce":2445326158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.13:6802","nonce":2445326158},{"type":"v1","addr":"10.248.50.13:6803","nonce":2445326158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.13:6806","nonce":2445326158},{"type":"v1","addr":"10.248.50.13:6807","nonce":2445326158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.13:6804","nonce":2445326158},{"type":"v1","addr":"10.248.50.13:6805","nonce":2445326158}]},"public_addr":"10.248.50.13:6801/2445326158","cluster_addr":"10.248.50.13:6803/2445326158","heartbeat_back_addr":"10.248.50.13:6807/2445326158","heartbeat_front_addr":"10.248.50.13:6805/2445326158","state":["exists","up"]},{"osd":5,"uuid":"c0bded03-f49f-4e53-bd04-8d280177153c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.11:6800","nonce":2167751379},{"type":"v1","addr":"10.248.48.11:6801","nonce":2167751379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.11:6802","nonce":2167751379},{"type":"v1","addr":"10.248.48.11:6803","nonce":2167751379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.11:6806","nonce":2167751379},{"type":"v1","addr":"10.248.48.11:6807","nonce":2167751379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.11:6804","nonce":2167751379},{"type":"v1","addr":"10.248.48.11:6805","nonce":2167751379}]},"public_addr":"10.248.48.11:6801/2167751379","cluster_addr":"10.248.48.11:6803/2167751379","heartbeat_back_addr":"10.248.48.11:6807/2167751379","heartbeat_front_addr":"10.248.48.11:6805/2167751379","state":["exists","up"]},{"osd":6,"uuid":"b9f130db-343c-49ea-8a12-508c0c51d79b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.14:6800","nonce":3718528490},{"type":"v1","addr":"10.248.50.14:6801","nonce":3718528490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.14:6802","nonce":3718528490},{"type":"v1","addr":"10.248.50.14:6803","nonce":3718528490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.14:6806","nonce":3718528490},{"type":"v1","addr":"10.248.50.14:6807","nonce":3718528490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.14:6804","nonce":3718528490},{"type":"v1","addr":"10.248.50.14:6805","nonce":3718528490}]},"public_addr":"10.248.50.14:6801/3718528490","cluster_addr":"10.248.50.14:6803/3718528490","heartbeat_back_addr":"10.248.50.14:6807/3718528490","heartbeat_front_addr":"10.248.50.14:6805/3718528490","state":["exists","up"]},{"osd":7,"uuid":"514523ef-1d10-4e0b-a9c5-a6b789f988ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.13:6800","nonce":561302417},{"type":"v1","addr":"10.248.48.13:6801","nonce":561302417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.13:6802","nonce":561302417},{"type":"v1","addr":"10.248.48.13:6803","nonce":561302417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.13:6806","nonce":561302417},{"type":"v1","addr":"10.248.48.13:6807","nonce":561302417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.13:6804","nonce":561302417},{"type":"v1","addr":"10.248.48.13:6805","nonce":561302417}]},"public_addr":"10.248.48.13:6801/561302417","cluster_addr":"10.248.48.13:6803/561302417","heartbeat_back_addr":"10.248.48.13:6807/561302417","heartbeat_front_addr":"10.248.48.13:6805/561302417","state":["exists","up"]},{"osd":8,"uuid":"ba3091ce-f7a7-43f1-907e-f60ca7952110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.14:6800","nonce":961922660},{"type":"v1","addr":"10.248.48.14:6801","nonce":961922660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.14:6802","nonce":961922660},{"type":"v1","addr":"10.248.48.14:6803","nonce":961922660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.14:6806","nonce":961922660},{"type":"v1","addr":"10.248.48.14:6807","nonce":961922660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.14:6804","nonce":961922660},{"type":"v1","addr":"10.248.48.14:6805","nonce":961922660}]},"public_addr":"10.248.48.14:6801/961922660","cluster_addr":"10.248.48.14:6803/961922660","heartbeat_back_addr":"10.248.48.14:6807/961922660","heartbeat_front_addr":"10.248.48.14:6805/961922660","state":["exists","up"]},{"osd":9,"uuid":"6a1dd956-9931-411c-ad1d-4a83e702e703","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.15:6800","nonce":3919597058},{"type":"v1","addr":"10.248.48.15:6801","nonce":3919597058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.15:6802","nonce":3919597058},{"type":"v1","addr":"10.248.48.15:6803","nonce":3919597058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.15:6806","nonce":3919597058},{"type":"v1","addr":"10.248.48.15:6807","nonce":3919597058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.15:6804","nonce":3919597058},{"type":"v1","addr":"10.248.48.15:6805","nonce":3919597058}]},"public_addr":"10.248.48.15:6801/3919597058","cluster_addr":"10.248.48.15:6803/3919597058","heartbeat_back_addr":"10.248.48.15:6807/3919597058","heartbeat_front_addr":"10.248.48.15:6805/3919597058","state":["exists","up"]},{"osd":10,"uuid":"f06ae5c1-223a-424e-be38-2d6dd7de07f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.18:6800","nonce":1792565641},{"type":"v1","addr":"10.248.50.18:6801","nonce":1792565641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.18:6802","nonce":1792565641},{"type":"v1","addr":"10.248.50.18:6803","nonce":1792565641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.18:6806","nonce":1792565641},{"type":"v1","addr":"10.248.50.18:6807","nonce":1792565641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.18:6804","nonce":1792565641},{"type":"v1","addr":"10.248.50.18:6805","nonce":1792565641}]},"public_addr":"10.248.50.18:6801/1792565641","cluster_addr":"10.248.50.18:6803/1792565641","heartbeat_back_addr":"10.248.50.18:6807/1792565641","heartbeat_front_addr":"10.248.50.18:6805/1792565641","state":["exists","up"]},{"osd":11,"uuid":"e208cea4-d52f-4b0f-baad-da47c6844bf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.17:6800","nonce":2349665581},{"type":"v1","addr":"10.248.50.17:6801","nonce":2349665581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.17:6802","nonce":2349665581},{"type":"v1","addr":"10.248.50.17:6803","nonce":2349665581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.17:6806","nonce":2349665581},{"type":"v1","addr":"10.248.50.17:6807","nonce":2349665581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.17:6804","nonce":2349665581},{"type":"v1","addr":"10.248.50.17:6805","nonce":2349665581}]},"public_addr":"10.248.50.17:6801/2349665581","cluster_addr":"10.248.50.17:6803/2349665581","heartbeat_back_addr":"10.248.50.17:6807/2349665581","heartbeat_front_addr":"10.248.50.17:6805/2349665581","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:58:58.329848+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:58:56.055790+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:58:58.232279+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:58:55.949487+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:58:47.433907+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:59:10.833121+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:58:56.802957+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:59:10.930410+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:59:10.980307+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:59:12.217528+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:59:53.384240+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-20T14:59:51.964883+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.248.48.5:0/1750087674":"2022-03-21T14:57:16.309841+0000","10.248.48.5:0/3041300154":"2022-03-21T14:57:16.309841+0000","10.248.48.5:0/1222231597":"2022-03-21T14:57:16.309841+0000","10.248.48.5:6801/4177463852":"2022-03-21T14:57:16.309841+0000","10.248.48.5:6800/4177463852":"2022-03-21T14:57:16.309841+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-20T15:00:11.026 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph tell osd.0 flush_pg_stats 2022-03-20T15:00:11.027 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph tell osd.1 flush_pg_stats 2022-03-20T15:00:11.028 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph tell osd.2 flush_pg_stats 2022-03-20T15:00:11.029 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph tell osd.3 flush_pg_stats 2022-03-20T15:00:11.029 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph tell osd.4 flush_pg_stats 2022-03-20T15:00:11.030 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph tell osd.5 flush_pg_stats 2022-03-20T15:00:11.030 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph tell osd.6 flush_pg_stats 2022-03-20T15:00:11.031 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph tell osd.7 flush_pg_stats 2022-03-20T15:00:11.032 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph tell osd.8 flush_pg_stats 2022-03-20T15:00:11.032 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph tell osd.9 flush_pg_stats 2022-03-20T15:00:11.033 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph tell osd.10 flush_pg_stats 2022-03-20T15:00:11.033 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph tell osd.11 flush_pg_stats 2022-03-20T15:00:11.556 INFO:teuthology.orchestra.run.smithi007.stdout:120259084302 2022-03-20T15:00:11.556 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd last-stat-seq osd.8 2022-03-20T15:00:11.962 INFO:teuthology.orchestra.run.smithi007.stdout:98784247824 2022-03-20T15:00:11.963 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd last-stat-seq osd.0 2022-03-20T15:00:11.966 INFO:teuthology.orchestra.run.smithi007.stdout:98784247824 2022-03-20T15:00:11.966 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd last-stat-seq osd.2 2022-03-20T15:00:11.970 INFO:teuthology.orchestra.run.smithi007.stdout:124554051598 2022-03-20T15:00:11.971 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd last-stat-seq osd.9 2022-03-20T15:00:11.972 INFO:teuthology.orchestra.run.smithi007.stdout:94489280529 2022-03-20T15:00:11.973 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd last-stat-seq osd.6 2022-03-20T15:00:11.974 INFO:teuthology.orchestra.run.smithi007.stdout:120259084301 2022-03-20T15:00:11.975 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd last-stat-seq osd.5 2022-03-20T15:00:11.979 INFO:teuthology.orchestra.run.smithi007.stdout:90194313233 2022-03-20T15:00:11.979 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd last-stat-seq osd.1 2022-03-20T15:00:11.983 INFO:teuthology.orchestra.run.smithi007.stdout:90194313232 2022-03-20T15:00:11.983 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd last-stat-seq osd.3 2022-03-20T15:00:11.987 INFO:teuthology.orchestra.run.smithi007.stdout:120259084301 2022-03-20T15:00:11.987 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd last-stat-seq osd.7 2022-03-20T15:00:12.025 INFO:teuthology.orchestra.run.smithi007.stdout:154618822662 2022-03-20T15:00:12.025 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd last-stat-seq osd.10 2022-03-20T15:00:12.043 INFO:teuthology.orchestra.run.smithi007.stdout:150323855367 2022-03-20T15:00:12.044 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd last-stat-seq osd.11 2022-03-20T15:00:12.178 INFO:teuthology.orchestra.run.smithi007.stdout:73014444051 2022-03-20T15:00:12.179 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd last-stat-seq osd.4 2022-03-20T15:00:12.448 INFO:teuthology.orchestra.run.smithi007.stdout:120259084301 2022-03-20T15:00:12.523 INFO:tasks.rook.ceph_manager.ceph:need seq 120259084302 got 120259084301 for osd.8 2022-03-20T15:00:12.740 INFO:teuthology.orchestra.run.smithi007.stdout:90194313232 2022-03-20T15:00:12.851 INFO:teuthology.orchestra.run.smithi007.stdout:98784247823 2022-03-20T15:00:12.859 INFO:teuthology.orchestra.run.smithi007.stdout:120259084300 2022-03-20T15:00:12.865 INFO:tasks.rook.ceph_manager.ceph:need seq 90194313233 got 90194313232 for osd.1 2022-03-20T15:00:12.909 INFO:tasks.rook.ceph_manager.ceph:need seq 120259084301 got 120259084300 for osd.5 2022-03-20T15:00:12.912 INFO:teuthology.orchestra.run.smithi007.stdout:98784247824 2022-03-20T15:00:12.913 INFO:tasks.rook.ceph_manager.ceph:need seq 98784247824 got 98784247823 for osd.2 2022-03-20T15:00:12.946 INFO:tasks.rook.ceph_manager.ceph:need seq 98784247824 got 98784247824 for osd.0 2022-03-20T15:00:12.947 DEBUG:teuthology.parallel:result is None 2022-03-20T15:00:13.026 INFO:teuthology.orchestra.run.smithi007.stdout:94489280528 2022-03-20T15:00:13.042 INFO:teuthology.orchestra.run.smithi007.stdout:154618822661 2022-03-20T15:00:13.048 INFO:tasks.rook.ceph_manager.ceph:need seq 94489280529 got 94489280528 for osd.6 2022-03-20T15:00:13.060 INFO:teuthology.orchestra.run.smithi007.stdout:124554051597 2022-03-20T15:00:13.065 INFO:tasks.rook.ceph_manager.ceph:need seq 154618822662 got 154618822661 for osd.10 2022-03-20T15:00:13.088 INFO:tasks.rook.ceph_manager.ceph:need seq 124554051598 got 124554051597 for osd.9 2022-03-20T15:00:13.183 INFO:teuthology.orchestra.run.smithi007.stdout:150323855366 2022-03-20T15:00:13.202 INFO:tasks.rook.ceph_manager.ceph:need seq 150323855367 got 150323855366 for osd.11 2022-03-20T15:00:13.210 INFO:teuthology.orchestra.run.smithi007.stdout:120259084300 2022-03-20T15:00:13.225 INFO:teuthology.orchestra.run.smithi007.stdout:90194313231 2022-03-20T15:00:13.229 INFO:tasks.rook.ceph_manager.ceph:need seq 120259084301 got 120259084300 for osd.7 2022-03-20T15:00:13.248 INFO:tasks.rook.ceph_manager.ceph:need seq 90194313232 got 90194313231 for osd.3 2022-03-20T15:00:13.269 INFO:teuthology.orchestra.run.smithi007.stdout:73014444050 2022-03-20T15:00:13.289 INFO:tasks.rook.ceph_manager.ceph:need seq 73014444051 got 73014444050 for osd.4 2022-03-20T15:00:13.524 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd last-stat-seq osd.8 2022-03-20T15:00:13.866 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd last-stat-seq osd.1 2022-03-20T15:00:13.911 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd last-stat-seq osd.5 2022-03-20T15:00:13.914 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd last-stat-seq osd.2 2022-03-20T15:00:14.049 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd last-stat-seq osd.6 2022-03-20T15:00:14.066 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd last-stat-seq osd.10 2022-03-20T15:00:14.089 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd last-stat-seq osd.9 2022-03-20T15:00:14.098 INFO:teuthology.orchestra.run.smithi007.stdout:120259084302 2022-03-20T15:00:14.203 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd last-stat-seq osd.11 2022-03-20T15:00:14.207 INFO:tasks.rook.ceph_manager.ceph:need seq 120259084302 got 120259084302 for osd.8 2022-03-20T15:00:14.208 DEBUG:teuthology.parallel:result is None 2022-03-20T15:00:14.231 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd last-stat-seq osd.7 2022-03-20T15:00:14.250 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd last-stat-seq osd.3 2022-03-20T15:00:14.290 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph osd last-stat-seq osd.4 2022-03-20T15:00:14.649 INFO:teuthology.orchestra.run.smithi007.stdout:90194313233 2022-03-20T15:00:14.718 INFO:tasks.rook.ceph_manager.ceph:need seq 90194313233 got 90194313233 for osd.1 2022-03-20T15:00:14.718 DEBUG:teuthology.parallel:result is None 2022-03-20T15:00:14.761 INFO:teuthology.orchestra.run.smithi007.stdout:120259084301 2022-03-20T15:00:14.808 INFO:tasks.rook.ceph_manager.ceph:need seq 120259084301 got 120259084301 for osd.5 2022-03-20T15:00:14.808 DEBUG:teuthology.parallel:result is None 2022-03-20T15:00:14.828 INFO:teuthology.orchestra.run.smithi007.stdout:98784247825 2022-03-20T15:00:14.858 INFO:tasks.rook.ceph_manager.ceph:need seq 98784247824 got 98784247825 for osd.2 2022-03-20T15:00:14.859 DEBUG:teuthology.parallel:result is None 2022-03-20T15:00:14.889 INFO:teuthology.orchestra.run.smithi007.stdout:154618822662 2022-03-20T15:00:14.903 INFO:teuthology.orchestra.run.smithi007.stdout:94489280529 2022-03-20T15:00:14.922 INFO:tasks.rook.ceph_manager.ceph:need seq 154618822662 got 154618822662 for osd.10 2022-03-20T15:00:14.922 DEBUG:teuthology.parallel:result is None 2022-03-20T15:00:14.926 INFO:tasks.rook.ceph_manager.ceph:need seq 94489280529 got 94489280529 for osd.6 2022-03-20T15:00:14.927 DEBUG:teuthology.parallel:result is None 2022-03-20T15:00:15.032 INFO:teuthology.orchestra.run.smithi007.stdout:150323855367 2022-03-20T15:00:15.053 INFO:tasks.rook.ceph_manager.ceph:need seq 150323855367 got 150323855367 for osd.11 2022-03-20T15:00:15.054 DEBUG:teuthology.parallel:result is None 2022-03-20T15:00:15.109 INFO:teuthology.orchestra.run.smithi007.stdout:124554051598 2022-03-20T15:00:15.136 INFO:tasks.rook.ceph_manager.ceph:need seq 124554051598 got 124554051598 for osd.9 2022-03-20T15:00:15.137 DEBUG:teuthology.parallel:result is None 2022-03-20T15:00:15.176 INFO:teuthology.orchestra.run.smithi007.stdout:120259084301 2022-03-20T15:00:15.197 INFO:tasks.rook.ceph_manager.ceph:need seq 120259084301 got 120259084301 for osd.7 2022-03-20T15:00:15.198 DEBUG:teuthology.parallel:result is None 2022-03-20T15:00:15.242 INFO:teuthology.orchestra.run.smithi007.stdout:73014444052 2022-03-20T15:00:15.254 INFO:teuthology.orchestra.run.smithi007.stdout:90194313232 2022-03-20T15:00:15.261 INFO:tasks.rook.ceph_manager.ceph:need seq 73014444051 got 73014444052 for osd.4 2022-03-20T15:00:15.262 DEBUG:teuthology.parallel:result is None 2022-03-20T15:00:15.279 INFO:tasks.rook.ceph_manager.ceph:need seq 90194313232 got 90194313232 for osd.3 2022-03-20T15:00:15.279 DEBUG:teuthology.parallel:result is None 2022-03-20T15:00:15.280 INFO:tasks.rook.ceph_manager.ceph:waiting for clean 2022-03-20T15:00:15.280 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph pg dump --format=json 2022-03-20T15:00:15.816 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:00:15.818 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2022-03-20T15:00:15.836 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":117,"stamp":"2022-03-20T15:00:15.552950+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":67000,"kb_used_data":5976,"kb_used_omap":0,"kb_used_meta":60928,"kb_avail":1124874824,"statfs":{"total":1151940427776,"available":1151871819776,"internally_reserved":0,"allocated":6119424,"data_stored":3510216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":62390272},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003838"},"pg_stats":[{"pgid":"1.0","version":"25'117","reported_seq":202,"reported_epoch":36,"state":"active+clean","last_fresh":"2022-03-20T14:59:54.910725+0000","last_change":"2022-03-20T14:59:18.116859+0000","last_active":"2022-03-20T14:59:54.910725+0000","last_peered":"2022-03-20T14:59:54.910725+0000","last_clean":"2022-03-20T14:59:54.910725+0000","last_became_active":"2022-03-20T14:59:18.116274+0000","last_became_peered":"2022-03-20T14:59:18.116274+0000","last_unstale":"2022-03-20T14:59:54.910725+0000","last_undegraded":"2022-03-20T14:59:54.910725+0000","last_fullsized":"2022-03-20T14:59:54.910725+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-20T14:59:00.069883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-20T14:59:00.069883+0000","last_clean_scrub_stamp":"2022-03-20T14:59:00.069883+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-21T15:38:17.245813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":10,"up_from":36,"seq":154618822662,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5240,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739912,"statfs":{"total":95995035648,"available":95989669888,"internally_reserved":0,"allocated":376832,"data_stored":160286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7430000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0389999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.911}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3620000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8169999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.954}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.702}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7769999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0129999999999999}]}]},{"osd":11,"up_from":35,"seq":150323855368,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5240,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739912,"statfs":{"total":95995035648,"available":95989669888,"internally_reserved":0,"allocated":376832,"data_stored":160286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.647}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7270000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.772}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.294}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.861}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1749999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2160000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0979999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8089999999999999}]}]},{"osd":9,"up_from":29,"seq":124554051599,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5304,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739848,"statfs":{"total":95995035648,"available":95989604352,"internally_reserved":0,"allocated":376832,"data_stored":160286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.486}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1619999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6830000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.627}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0790000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2809999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3939999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3799999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5739999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0990000000000002}]}]},{"osd":8,"up_from":28,"seq":120259084303,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5304,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739848,"statfs":{"total":95995035648,"available":95989604352,"internally_reserved":0,"allocated":376832,"data_stored":160286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0469999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5509999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.629}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.909}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.248}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.875}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3220000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8009999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.98}]}]},{"osd":7,"up_from":28,"seq":120259084302,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5304,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739848,"statfs":{"total":95995035648,"available":95989604352,"internally_reserved":0,"allocated":376832,"data_stored":160286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 20 15:00:13 2022","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"min":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":1.968},{"interface":"front","average":{"1min":1.468,"5min":1.468,"15min":1.468},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":2.1829999999999998,"5min":2.1829999999999998,"15min":2.1829999999999998},"last":1.637}]},{"osd":1,"last update":"Sun Mar 20 15:00:13 2022","interfaces":[{"interface":"back","average":{"1min":1.5740000000000001,"5min":1.5740000000000001,"15min":1.5740000000000001},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"max":{"1min":2.9940000000000002,"5min":2.9940000000000002,"15min":2.9940000000000002},"last":2.0510000000000002},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"min":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"max":{"1min":2.141,"5min":2.141,"15min":2.141},"last":2.0099999999999998}]},{"osd":2,"last update":"Sun Mar 20 15:00:13 2022","interfaces":[{"interface":"back","average":{"1min":1.6120000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"min":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"max":{"1min":3.133,"5min":3.133,"15min":3.133},"last":2.1150000000000002},{"interface":"front","average":{"1min":1.542,"5min":1.542,"15min":1.542},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":2.3090000000000002,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.7190000000000001}]},{"osd":3,"last update":"Sun Mar 20 15:00:13 2022","interfaces":[{"interface":"back","average":{"1min":1.613,"5min":1.613,"15min":1.613},"min":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"max":{"1min":2.5899999999999999,"5min":2.5899999999999999,"15min":2.5899999999999999},"last":1.9350000000000001},{"interface":"front","average":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":1.9490000000000001},"min":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"max":{"1min":4.6180000000000003,"5min":4.6180000000000003,"15min":4.6180000000000003},"last":2.1859999999999999}]},{"osd":4,"last update":"Sun Mar 20 15:00:13 2022","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.5,"15min":1.5},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":2.048,"5min":2.048,"15min":2.048},"last":1.752},{"interface":"front","average":{"1min":1.655,"5min":1.655,"15min":1.655},"min":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":2.8050000000000002,"5min":2.8050000000000002,"15min":2.8050000000000002},"last":1.893}]},{"osd":5,"last update":"Sun Mar 20 15:00:13 2022","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.4099999999999999,"15min":1.4099999999999999},"min":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":2.23,"5min":2.23,"15min":2.23},"last":1.206},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"min":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"max":{"1min":2.0659999999999998,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":1.407}]},{"osd":6,"last update":"Sun Mar 20 15:00:13 2022","interfaces":[{"interface":"back","average":{"1min":1.724,"5min":1.724,"15min":1.724},"min":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"max":{"1min":3.0390000000000001,"5min":3.0390000000000001,"15min":3.0390000000000001},"last":1.5069999999999999},{"interface":"front","average":{"1min":1.698,"5min":1.698,"15min":1.698},"min":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"max":{"1min":2.1970000000000001,"5min":2.1970000000000001,"15min":2.1970000000000001},"last":1.5649999999999999}]},{"osd":8,"last update":"Sun Mar 20 15:00:13 2022","interfaces":[{"interface":"back","average":{"1min":1.847,"5min":1.847,"15min":1.847},"min":{"1min":0.997,"5min":0.997,"15min":0.997},"max":{"1min":2.927,"5min":2.927,"15min":2.927},"last":1.7929999999999999},{"interface":"front","average":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.8069999999999999},"min":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"max":{"1min":2.3279999999999998,"5min":2.3279999999999998,"15min":2.3279999999999998},"last":1.2470000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.333}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.306}]}]},{"osd":5,"up_from":28,"seq":120259084302,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5888,"kb_used_data":888,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739264,"statfs":{"total":95995035648,"available":95989006336,"internally_reserved":0,"allocated":909312,"data_stored":689214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0009999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.121}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0489999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.349}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5660000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5780000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.351}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.833}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8049999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3929999999999998}]}]},{"osd":3,"up_from":21,"seq":90194313233,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5368,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739784,"statfs":{"total":95995035648,"available":95989538816,"internally_reserved":0,"allocated":376832,"data_stored":160286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 20 15:00:02 2022","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.052,"15min":1.052},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.756,"5min":1.756,"15min":1.756},"last":0.88900000000000001},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":2.367,"5min":2.367,"15min":2.367},"last":0.95699999999999996}]},{"osd":1,"last update":"Sun Mar 20 15:00:00 2022","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.988,"5min":1.988,"15min":1.988},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.452,"5min":2.452,"15min":2.452},"last":1.113}]},{"osd":2,"last update":"Sun Mar 20 15:00:02 2022","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":1.659,"15min":1.659},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":2.3029999999999999,"5min":2.3029999999999999,"15min":2.3029999999999999},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.669,"5min":1.669,"15min":1.669},"min":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":3.996,"5min":3.996,"15min":3.996},"last":2.1970000000000001}]},{"osd":4,"last update":"Sun Mar 20 15:00:00 2022","interfaces":[{"interface":"back","average":{"1min":1.593,"5min":1.593,"15min":1.593},"min":{"1min":1.008,"5min":1.008,"15min":1.008},"max":{"1min":2.5049999999999999,"5min":2.5049999999999999,"15min":2.5049999999999999},"last":1.226},{"interface":"front","average":{"1min":1.7889999999999999,"5min":1.7889999999999999,"15min":1.7889999999999999},"min":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"max":{"1min":2.766,"5min":2.766,"15min":2.766},"last":1.494}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2879999999999998}]},{"osd":6,"last update":"Sun Mar 20 15:00:02 2022","interfaces":[{"interface":"back","average":{"1min":1.885,"5min":1.885,"15min":1.885},"min":{"1min":1.081,"5min":1.081,"15min":1.081},"max":{"1min":2.8820000000000001,"5min":2.8820000000000001,"15min":2.8820000000000001},"last":1.716},{"interface":"front","average":{"1min":1.897,"5min":1.897,"15min":1.897},"min":{"1min":1.153,"5min":1.153,"15min":1.153},"max":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"last":2.3700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9650000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4059999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.585}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.915}]}]},{"osd":4,"up_from":17,"seq":73014444052,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5880,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93739272,"statfs":{"total":95995035648,"available":95989014528,"internally_reserved":0,"allocated":376832,"data_stored":160286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 20 15:00:05 2022","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.101,"15min":1.101},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.724,"5min":1.724,"15min":1.724},"last":1.0660000000000001},{"interface":"front","average":{"1min":1.24,"5min":1.24,"15min":1.24},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.998,"5min":1.998,"15min":1.998},"last":1.1950000000000001}]},{"osd":1,"last update":"Sun Mar 20 14:59:59 2022","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.321,"15min":1.321},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":2.0249999999999999,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":1.5129999999999999},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":1.667,"5min":1.667,"15min":1.667},"last":1.3500000000000001}]},{"osd":2,"last update":"Sun Mar 20 15:00:05 2022","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.327,"15min":1.327},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":2.0659999999999998,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.355,"5min":1.355,"15min":1.355},"min":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"max":{"1min":3.6349999999999998,"5min":3.6349999999999998,"15min":3.6349999999999998},"last":1.4079999999999999}]},{"osd":3,"last update":"Sun Mar 20 14:59:59 2022","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.496,"15min":1.496},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.1110000000000002,"5min":2.1110000000000002,"15min":2.1110000000000002},"last":1.1120000000000001},{"interface":"front","average":{"1min":1.494,"5min":1.494,"15min":1.494},"min":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"max":{"1min":3.9089999999999998,"5min":3.9089999999999998,"15min":3.9089999999999998},"last":1.484}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.571}]},{"osd":6,"last update":"Sun Mar 20 15:00:05 2022","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":2.694,"5min":2.694,"15min":2.694},"last":1.157},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":2.371,"5min":2.371,"15min":2.371},"last":1.4370000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5580000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.915}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.956}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.681}]}]},{"osd":1,"up_from":21,"seq":90194313234,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6400,"kb_used_data":888,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738752,"statfs":{"total":95995035648,"available":95988482048,"internally_reserved":0,"allocated":909312,"data_stored":689214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 20 15:00:04 2022","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.996,"15min":0.996},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.7,"5min":1.7,"15min":1.7},"last":0.61299999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":0.57499999999999996}]},{"osd":2,"last update":"Sun Mar 20 15:00:04 2022","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.117,"15min":1.117},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":4.3479999999999999},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"min":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":2.1019999999999999,"5min":2.1019999999999999,"15min":2.1019999999999999},"last":1.4019999999999999}]},{"osd":3,"last update":"Sun Mar 20 14:59:59 2022","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":2.3519999999999999,"5min":2.3519999999999999,"15min":2.3519999999999999},"last":4.4059999999999997},{"interface":"front","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":2.2879999999999998,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":0.64800000000000002}]},{"osd":4,"last update":"Sun Mar 20 14:59:59 2022","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":2.0710000000000002,"5min":2.0710000000000002,"15min":2.0710000000000002},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"min":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":2.3450000000000002,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":0.82999999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":6,"last update":"Sun Mar 20 15:00:04 2022","interfaces":[{"interface":"back","average":{"1min":1.7150000000000001,"5min":1.7150000000000001,"15min":1.7150000000000001},"min":{"1min":1.115,"5min":1.115,"15min":1.115},"max":{"1min":2.4249999999999998,"5min":2.4249999999999998,"15min":2.4249999999999998},"last":1.103},{"interface":"front","average":{"1min":1.704,"5min":1.704,"15min":1.704},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":2.306,"5min":2.306,"15min":2.306},"last":0.88600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0680000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.431}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.327}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.347}]}]},{"osd":6,"up_from":22,"seq":94489280529,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5368,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739784,"statfs":{"total":95995035648,"available":95989538816,"internally_reserved":0,"allocated":376832,"data_stored":160286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 20 15:00:04 2022","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.7310000000000001,"5min":1.7310000000000001,"15min":1.7310000000000001},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.288,"5min":1.288,"15min":1.288},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":1.8959999999999999,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":1.274}]},{"osd":1,"last update":"Sun Mar 20 15:00:02 2022","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"min":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"max":{"1min":2.1379999999999999,"5min":2.1379999999999999,"15min":2.1379999999999999},"last":1.6679999999999999},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"min":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"max":{"1min":1.976,"5min":1.976,"15min":1.976},"last":1.9450000000000001}]},{"osd":2,"last update":"Sun Mar 20 15:00:04 2022","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"min":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":2.2440000000000002,"5min":2.2440000000000002,"15min":2.2440000000000002},"last":1.2490000000000001},{"interface":"front","average":{"1min":1.405,"5min":1.405,"15min":1.405},"min":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"max":{"1min":2.0790000000000002,"5min":2.0790000000000002,"15min":2.0790000000000002},"last":2.1219999999999999}]},{"osd":3,"last update":"Sun Mar 20 15:00:02 2022","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":2.0030000000000001,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":1.6000000000000001},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":2.234,"5min":2.234,"15min":2.234},"last":2.278}]},{"osd":4,"last update":"Sun Mar 20 15:00:02 2022","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.843,"5min":1.843,"15min":1.843},"last":1.198},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.46,"5min":1.46,"15min":1.46},"last":0.94099999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.714}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0489999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8280000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8149999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1479999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247825,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5368,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739784,"statfs":{"total":95995035648,"available":95989538816,"internally_reserved":0,"allocated":376832,"data_stored":160286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Mar 20 15:00:05 2022","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.7589999999999999,"5min":1.7589999999999999,"15min":1.7589999999999999},"last":1.9239999999999999},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":2.1760000000000002,"5min":2.1760000000000002,"15min":2.1760000000000002},"last":1.427}]},{"osd":2,"last update":"Sun Mar 20 15:00:05 2022","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.228,"15min":1.228},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.853,"5min":1.853,"15min":1.853},"last":1.5029999999999999},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.871,"5min":1.871,"15min":1.871},"last":1.0640000000000001}]},{"osd":3,"last update":"Sun Mar 20 15:00:05 2022","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.254,"15min":1.254},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.909,"5min":1.909,"15min":1.909},"last":2.0230000000000001},{"interface":"front","average":{"1min":1.601,"5min":1.601,"15min":1.601},"min":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"max":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"last":2.3279999999999998}]},{"osd":4,"last update":"Sun Mar 20 15:00:05 2022","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.409,"15min":1.409},"min":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":1.837,"5min":1.837,"15min":1.837},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"min":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"max":{"1min":2.875,"5min":2.875,"15min":2.875},"last":1.8180000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5569999999999999}]},{"osd":6,"last update":"Sun Mar 20 15:00:05 2022","interfaces":[{"interface":"back","average":{"1min":1.663,"5min":1.663,"15min":1.663},"min":{"1min":1.141,"5min":1.141,"15min":1.141},"max":{"1min":2.2050000000000001,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":1.6439999999999999},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"min":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":1.9530000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6100000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6280000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8620000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8680000000000001}]}]},{"osd":2,"up_from":23,"seq":98784247825,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6336,"kb_used_data":888,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738816,"statfs":{"total":95995035648,"available":95988547584,"internally_reserved":0,"allocated":909312,"data_stored":689214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 20 15:00:04 2022","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.55,"5min":1.55,"15min":1.55},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":2.0680000000000001,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":1.079}]},{"osd":1,"last update":"Sun Mar 20 15:00:04 2022","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.3340000000000001,"5min":2.3340000000000001,"15min":2.3340000000000001},"last":0.95099999999999996},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.9239999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":1.623}]},{"osd":3,"last update":"Sun Mar 20 15:00:04 2022","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":1.502,"15min":1.502},"min":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":1.992,"5min":1.992,"15min":1.992},"last":1.2629999999999999},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":2.3980000000000001,"5min":2.3980000000000001,"15min":2.3980000000000001},"last":1.8300000000000001}]},{"osd":4,"last update":"Sun Mar 20 15:00:04 2022","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.532,"15min":1.532},"min":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"max":{"1min":2.52,"5min":2.52,"15min":2.52},"last":1.474},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"min":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":2.3519999999999999,"5min":2.3519999999999999,"15min":2.3519999999999999},"last":1.1819999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7549999999999999}]},{"osd":6,"last update":"Sun Mar 20 15:00:04 2022","interfaces":[{"interface":"back","average":{"1min":1.726,"5min":1.726,"15min":1.726},"min":{"1min":1.159,"5min":1.159,"15min":1.159},"max":{"1min":2.7370000000000001,"5min":2.7370000000000001,"15min":2.7370000000000001},"last":1.3799999999999999},{"interface":"front","average":{"1min":1.7350000000000001,"5min":1.7350000000000001,"15min":1.7350000000000001},"min":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"max":{"1min":2.7719999999999998,"5min":2.7719999999999998,"15min":2.7719999999999998},"last":1.556}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.514}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9890000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.865}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0619999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-20T15:00:15.839 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph pg dump --format=json 2022-03-20T15:00:16.388 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:00:16.390 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2022-03-20T15:00:16.408 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":117,"stamp":"2022-03-20T15:00:15.552950+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":67000,"kb_used_data":5976,"kb_used_omap":0,"kb_used_meta":60928,"kb_avail":1124874824,"statfs":{"total":1151940427776,"available":1151871819776,"internally_reserved":0,"allocated":6119424,"data_stored":3510216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":62390272},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003838"},"pg_stats":[{"pgid":"1.0","version":"25'117","reported_seq":202,"reported_epoch":36,"state":"active+clean","last_fresh":"2022-03-20T14:59:54.910725+0000","last_change":"2022-03-20T14:59:18.116859+0000","last_active":"2022-03-20T14:59:54.910725+0000","last_peered":"2022-03-20T14:59:54.910725+0000","last_clean":"2022-03-20T14:59:54.910725+0000","last_became_active":"2022-03-20T14:59:18.116274+0000","last_became_peered":"2022-03-20T14:59:18.116274+0000","last_unstale":"2022-03-20T14:59:54.910725+0000","last_undegraded":"2022-03-20T14:59:54.910725+0000","last_fullsized":"2022-03-20T14:59:54.910725+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-20T14:59:00.069883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-20T14:59:00.069883+0000","last_clean_scrub_stamp":"2022-03-20T14:59:00.069883+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-21T15:38:17.245813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":10,"up_from":36,"seq":154618822662,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5240,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739912,"statfs":{"total":95995035648,"available":95989669888,"internally_reserved":0,"allocated":376832,"data_stored":160286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7430000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0389999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.911}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3620000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8169999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.954}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.702}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7769999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0129999999999999}]}]},{"osd":11,"up_from":35,"seq":150323855368,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5240,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739912,"statfs":{"total":95995035648,"available":95989669888,"internally_reserved":0,"allocated":376832,"data_stored":160286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.647}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7270000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.772}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.294}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.861}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1749999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2160000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0979999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8089999999999999}]}]},{"osd":9,"up_from":29,"seq":124554051599,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5304,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739848,"statfs":{"total":95995035648,"available":95989604352,"internally_reserved":0,"allocated":376832,"data_stored":160286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.486}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1619999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6830000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.627}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0790000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2809999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3939999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3799999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5739999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0990000000000002}]}]},{"osd":8,"up_from":28,"seq":120259084303,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5304,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739848,"statfs":{"total":95995035648,"available":95989604352,"internally_reserved":0,"allocated":376832,"data_stored":160286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0469999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5509999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.629}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.909}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.248}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.875}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3220000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8009999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.98}]}]},{"osd":7,"up_from":28,"seq":120259084302,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5304,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739848,"statfs":{"total":95995035648,"available":95989604352,"internally_reserved":0,"allocated":376832,"data_stored":160286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 20 15:00:13 2022","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"min":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":1.968},{"interface":"front","average":{"1min":1.468,"5min":1.468,"15min":1.468},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":2.1829999999999998,"5min":2.1829999999999998,"15min":2.1829999999999998},"last":1.637}]},{"osd":1,"last update":"Sun Mar 20 15:00:13 2022","interfaces":[{"interface":"back","average":{"1min":1.5740000000000001,"5min":1.5740000000000001,"15min":1.5740000000000001},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"max":{"1min":2.9940000000000002,"5min":2.9940000000000002,"15min":2.9940000000000002},"last":2.0510000000000002},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"min":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"max":{"1min":2.141,"5min":2.141,"15min":2.141},"last":2.0099999999999998}]},{"osd":2,"last update":"Sun Mar 20 15:00:13 2022","interfaces":[{"interface":"back","average":{"1min":1.6120000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"min":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"max":{"1min":3.133,"5min":3.133,"15min":3.133},"last":2.1150000000000002},{"interface":"front","average":{"1min":1.542,"5min":1.542,"15min":1.542},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":2.3090000000000002,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.7190000000000001}]},{"osd":3,"last update":"Sun Mar 20 15:00:13 2022","interfaces":[{"interface":"back","average":{"1min":1.613,"5min":1.613,"15min":1.613},"min":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"max":{"1min":2.5899999999999999,"5min":2.5899999999999999,"15min":2.5899999999999999},"last":1.9350000000000001},{"interface":"front","average":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":1.9490000000000001},"min":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"max":{"1min":4.6180000000000003,"5min":4.6180000000000003,"15min":4.6180000000000003},"last":2.1859999999999999}]},{"osd":4,"last update":"Sun Mar 20 15:00:13 2022","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.5,"15min":1.5},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":2.048,"5min":2.048,"15min":2.048},"last":1.752},{"interface":"front","average":{"1min":1.655,"5min":1.655,"15min":1.655},"min":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":2.8050000000000002,"5min":2.8050000000000002,"15min":2.8050000000000002},"last":1.893}]},{"osd":5,"last update":"Sun Mar 20 15:00:13 2022","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.4099999999999999,"15min":1.4099999999999999},"min":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":2.23,"5min":2.23,"15min":2.23},"last":1.206},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"min":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"max":{"1min":2.0659999999999998,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":1.407}]},{"osd":6,"last update":"Sun Mar 20 15:00:13 2022","interfaces":[{"interface":"back","average":{"1min":1.724,"5min":1.724,"15min":1.724},"min":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"max":{"1min":3.0390000000000001,"5min":3.0390000000000001,"15min":3.0390000000000001},"last":1.5069999999999999},{"interface":"front","average":{"1min":1.698,"5min":1.698,"15min":1.698},"min":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"max":{"1min":2.1970000000000001,"5min":2.1970000000000001,"15min":2.1970000000000001},"last":1.5649999999999999}]},{"osd":8,"last update":"Sun Mar 20 15:00:13 2022","interfaces":[{"interface":"back","average":{"1min":1.847,"5min":1.847,"15min":1.847},"min":{"1min":0.997,"5min":0.997,"15min":0.997},"max":{"1min":2.927,"5min":2.927,"15min":2.927},"last":1.7929999999999999},{"interface":"front","average":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.8069999999999999},"min":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"max":{"1min":2.3279999999999998,"5min":2.3279999999999998,"15min":2.3279999999999998},"last":1.2470000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.333}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.306}]}]},{"osd":5,"up_from":28,"seq":120259084302,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5888,"kb_used_data":888,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739264,"statfs":{"total":95995035648,"available":95989006336,"internally_reserved":0,"allocated":909312,"data_stored":689214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0009999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.121}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0489999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.349}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5660000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5780000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.351}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.833}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8049999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3929999999999998}]}]},{"osd":3,"up_from":21,"seq":90194313233,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5368,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739784,"statfs":{"total":95995035648,"available":95989538816,"internally_reserved":0,"allocated":376832,"data_stored":160286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 20 15:00:02 2022","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.052,"15min":1.052},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.756,"5min":1.756,"15min":1.756},"last":0.88900000000000001},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":2.367,"5min":2.367,"15min":2.367},"last":0.95699999999999996}]},{"osd":1,"last update":"Sun Mar 20 15:00:00 2022","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.988,"5min":1.988,"15min":1.988},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.452,"5min":2.452,"15min":2.452},"last":1.113}]},{"osd":2,"last update":"Sun Mar 20 15:00:02 2022","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":1.659,"15min":1.659},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":2.3029999999999999,"5min":2.3029999999999999,"15min":2.3029999999999999},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.669,"5min":1.669,"15min":1.669},"min":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":3.996,"5min":3.996,"15min":3.996},"last":2.1970000000000001}]},{"osd":4,"last update":"Sun Mar 20 15:00:00 2022","interfaces":[{"interface":"back","average":{"1min":1.593,"5min":1.593,"15min":1.593},"min":{"1min":1.008,"5min":1.008,"15min":1.008},"max":{"1min":2.5049999999999999,"5min":2.5049999999999999,"15min":2.5049999999999999},"last":1.226},{"interface":"front","average":{"1min":1.7889999999999999,"5min":1.7889999999999999,"15min":1.7889999999999999},"min":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"max":{"1min":2.766,"5min":2.766,"15min":2.766},"last":1.494}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2879999999999998}]},{"osd":6,"last update":"Sun Mar 20 15:00:02 2022","interfaces":[{"interface":"back","average":{"1min":1.885,"5min":1.885,"15min":1.885},"min":{"1min":1.081,"5min":1.081,"15min":1.081},"max":{"1min":2.8820000000000001,"5min":2.8820000000000001,"15min":2.8820000000000001},"last":1.716},{"interface":"front","average":{"1min":1.897,"5min":1.897,"15min":1.897},"min":{"1min":1.153,"5min":1.153,"15min":1.153},"max":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"last":2.3700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9650000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4059999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.585}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.915}]}]},{"osd":4,"up_from":17,"seq":73014444052,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5880,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93739272,"statfs":{"total":95995035648,"available":95989014528,"internally_reserved":0,"allocated":376832,"data_stored":160286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 20 15:00:05 2022","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.101,"15min":1.101},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.724,"5min":1.724,"15min":1.724},"last":1.0660000000000001},{"interface":"front","average":{"1min":1.24,"5min":1.24,"15min":1.24},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.998,"5min":1.998,"15min":1.998},"last":1.1950000000000001}]},{"osd":1,"last update":"Sun Mar 20 14:59:59 2022","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.321,"15min":1.321},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":2.0249999999999999,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":1.5129999999999999},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":1.667,"5min":1.667,"15min":1.667},"last":1.3500000000000001}]},{"osd":2,"last update":"Sun Mar 20 15:00:05 2022","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.327,"15min":1.327},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":2.0659999999999998,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.355,"5min":1.355,"15min":1.355},"min":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"max":{"1min":3.6349999999999998,"5min":3.6349999999999998,"15min":3.6349999999999998},"last":1.4079999999999999}]},{"osd":3,"last update":"Sun Mar 20 14:59:59 2022","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.496,"15min":1.496},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.1110000000000002,"5min":2.1110000000000002,"15min":2.1110000000000002},"last":1.1120000000000001},{"interface":"front","average":{"1min":1.494,"5min":1.494,"15min":1.494},"min":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"max":{"1min":3.9089999999999998,"5min":3.9089999999999998,"15min":3.9089999999999998},"last":1.484}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.571}]},{"osd":6,"last update":"Sun Mar 20 15:00:05 2022","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":2.694,"5min":2.694,"15min":2.694},"last":1.157},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":2.371,"5min":2.371,"15min":2.371},"last":1.4370000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5580000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.915}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.956}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.681}]}]},{"osd":1,"up_from":21,"seq":90194313234,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6400,"kb_used_data":888,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738752,"statfs":{"total":95995035648,"available":95988482048,"internally_reserved":0,"allocated":909312,"data_stored":689214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 20 15:00:04 2022","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.996,"15min":0.996},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.7,"5min":1.7,"15min":1.7},"last":0.61299999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":0.57499999999999996}]},{"osd":2,"last update":"Sun Mar 20 15:00:04 2022","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.117,"15min":1.117},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":4.3479999999999999},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"min":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":2.1019999999999999,"5min":2.1019999999999999,"15min":2.1019999999999999},"last":1.4019999999999999}]},{"osd":3,"last update":"Sun Mar 20 14:59:59 2022","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":2.3519999999999999,"5min":2.3519999999999999,"15min":2.3519999999999999},"last":4.4059999999999997},{"interface":"front","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":2.2879999999999998,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":0.64800000000000002}]},{"osd":4,"last update":"Sun Mar 20 14:59:59 2022","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":2.0710000000000002,"5min":2.0710000000000002,"15min":2.0710000000000002},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"min":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":2.3450000000000002,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":0.82999999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":6,"last update":"Sun Mar 20 15:00:04 2022","interfaces":[{"interface":"back","average":{"1min":1.7150000000000001,"5min":1.7150000000000001,"15min":1.7150000000000001},"min":{"1min":1.115,"5min":1.115,"15min":1.115},"max":{"1min":2.4249999999999998,"5min":2.4249999999999998,"15min":2.4249999999999998},"last":1.103},{"interface":"front","average":{"1min":1.704,"5min":1.704,"15min":1.704},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":2.306,"5min":2.306,"15min":2.306},"last":0.88600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0680000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.431}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.327}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.347}]}]},{"osd":6,"up_from":22,"seq":94489280529,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5368,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739784,"statfs":{"total":95995035648,"available":95989538816,"internally_reserved":0,"allocated":376832,"data_stored":160286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 20 15:00:04 2022","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.7310000000000001,"5min":1.7310000000000001,"15min":1.7310000000000001},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.288,"5min":1.288,"15min":1.288},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":1.8959999999999999,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":1.274}]},{"osd":1,"last update":"Sun Mar 20 15:00:02 2022","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"min":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"max":{"1min":2.1379999999999999,"5min":2.1379999999999999,"15min":2.1379999999999999},"last":1.6679999999999999},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"min":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"max":{"1min":1.976,"5min":1.976,"15min":1.976},"last":1.9450000000000001}]},{"osd":2,"last update":"Sun Mar 20 15:00:04 2022","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"min":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":2.2440000000000002,"5min":2.2440000000000002,"15min":2.2440000000000002},"last":1.2490000000000001},{"interface":"front","average":{"1min":1.405,"5min":1.405,"15min":1.405},"min":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"max":{"1min":2.0790000000000002,"5min":2.0790000000000002,"15min":2.0790000000000002},"last":2.1219999999999999}]},{"osd":3,"last update":"Sun Mar 20 15:00:02 2022","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":2.0030000000000001,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":1.6000000000000001},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":2.234,"5min":2.234,"15min":2.234},"last":2.278}]},{"osd":4,"last update":"Sun Mar 20 15:00:02 2022","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.843,"5min":1.843,"15min":1.843},"last":1.198},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.46,"5min":1.46,"15min":1.46},"last":0.94099999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.714}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0489999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8280000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8149999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1479999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247825,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5368,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739784,"statfs":{"total":95995035648,"available":95989538816,"internally_reserved":0,"allocated":376832,"data_stored":160286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Mar 20 15:00:05 2022","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.7589999999999999,"5min":1.7589999999999999,"15min":1.7589999999999999},"last":1.9239999999999999},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":2.1760000000000002,"5min":2.1760000000000002,"15min":2.1760000000000002},"last":1.427}]},{"osd":2,"last update":"Sun Mar 20 15:00:05 2022","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.228,"15min":1.228},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.853,"5min":1.853,"15min":1.853},"last":1.5029999999999999},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.871,"5min":1.871,"15min":1.871},"last":1.0640000000000001}]},{"osd":3,"last update":"Sun Mar 20 15:00:05 2022","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.254,"15min":1.254},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.909,"5min":1.909,"15min":1.909},"last":2.0230000000000001},{"interface":"front","average":{"1min":1.601,"5min":1.601,"15min":1.601},"min":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"max":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"last":2.3279999999999998}]},{"osd":4,"last update":"Sun Mar 20 15:00:05 2022","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.409,"15min":1.409},"min":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":1.837,"5min":1.837,"15min":1.837},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"min":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"max":{"1min":2.875,"5min":2.875,"15min":2.875},"last":1.8180000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5569999999999999}]},{"osd":6,"last update":"Sun Mar 20 15:00:05 2022","interfaces":[{"interface":"back","average":{"1min":1.663,"5min":1.663,"15min":1.663},"min":{"1min":1.141,"5min":1.141,"15min":1.141},"max":{"1min":2.2050000000000001,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":1.6439999999999999},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"min":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":1.9530000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6100000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6280000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8620000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8680000000000001}]}]},{"osd":2,"up_from":23,"seq":98784247825,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6336,"kb_used_data":888,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738816,"statfs":{"total":95995035648,"available":95988547584,"internally_reserved":0,"allocated":909312,"data_stored":689214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 20 15:00:04 2022","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.55,"5min":1.55,"15min":1.55},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":2.0680000000000001,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":1.079}]},{"osd":1,"last update":"Sun Mar 20 15:00:04 2022","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.3340000000000001,"5min":2.3340000000000001,"15min":2.3340000000000001},"last":0.95099999999999996},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.9239999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":1.623}]},{"osd":3,"last update":"Sun Mar 20 15:00:04 2022","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":1.502,"15min":1.502},"min":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":1.992,"5min":1.992,"15min":1.992},"last":1.2629999999999999},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":2.3980000000000001,"5min":2.3980000000000001,"15min":2.3980000000000001},"last":1.8300000000000001}]},{"osd":4,"last update":"Sun Mar 20 15:00:04 2022","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.532,"15min":1.532},"min":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"max":{"1min":2.52,"5min":2.52,"15min":2.52},"last":1.474},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"min":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":2.3519999999999999,"5min":2.3519999999999999,"15min":2.3519999999999999},"last":1.1819999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7549999999999999}]},{"osd":6,"last update":"Sun Mar 20 15:00:04 2022","interfaces":[{"interface":"back","average":{"1min":1.726,"5min":1.726,"15min":1.726},"min":{"1min":1.159,"5min":1.159,"15min":1.159},"max":{"1min":2.7370000000000001,"5min":2.7370000000000001,"15min":2.7370000000000001},"last":1.3799999999999999},{"interface":"front","average":{"1min":1.7350000000000001,"5min":1.7350000000000001,"15min":1.7350000000000001},"min":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"max":{"1min":2.7719999999999998,"5min":2.7719999999999998,"15min":2.7719999999999998},"last":1.556}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.514}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9890000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.865}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0619999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-20T15:00:16.410 INFO:tasks.rook.ceph_manager.ceph:clean! 2022-03-20T15:00:16.411 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-03-20T15:00:16.411 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy 2022-03-20T15:00:16.412 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph health --format=json 2022-03-20T15:00:16.954 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:00:16.954 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-03-20T15:00:16.974 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy done 2022-03-20T15:00:16.975 INFO:tasks.rook:Rook complete, yielding 2022-03-20T15:00:16.975 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-20T15:00:16.995 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph -s 2022-03-20T15:00:17.582 INFO:teuthology.orchestra.run.smithi007.stdout: cluster: 2022-03-20T15:00:17.583 INFO:teuthology.orchestra.run.smithi007.stdout: id: ea418f6e-0cdf-45ac-b31b-73dc6739a8f8 2022-03-20T15:00:17.583 INFO:teuthology.orchestra.run.smithi007.stdout: health: HEALTH_OK 2022-03-20T15:00:17.583 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:00:17.584 INFO:teuthology.orchestra.run.smithi007.stdout: services: 2022-03-20T15:00:17.584 INFO:teuthology.orchestra.run.smithi007.stdout: mon: 3 daemons, quorum a,b,c (age 3m) 2022-03-20T15:00:17.584 INFO:teuthology.orchestra.run.smithi007.stdout: mgr: a(active, since 3m) 2022-03-20T15:00:17.584 INFO:teuthology.orchestra.run.smithi007.stdout: osd: 12 osds: 12 up (since 22s), 12 in (since 108s) 2022-03-20T15:00:17.585 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:00:17.585 INFO:teuthology.orchestra.run.smithi007.stdout: data: 2022-03-20T15:00:17.585 INFO:teuthology.orchestra.run.smithi007.stdout: pools: 1 pools, 1 pgs 2022-03-20T15:00:17.585 INFO:teuthology.orchestra.run.smithi007.stdout: objects: 2 objects, 577 KiB 2022-03-20T15:00:17.586 INFO:teuthology.orchestra.run.smithi007.stdout: usage: 65 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-20T15:00:17.586 INFO:teuthology.orchestra.run.smithi007.stdout: pgs: 1 active+clean 2022-03-20T15:00:17.586 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:00:17.602 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch status 2022-03-20T15:00:18.333 INFO:teuthology.orchestra.run.smithi007.stdout:Backend: rook 2022-03-20T15:00:18.333 INFO:teuthology.orchestra.run.smithi007.stdout:Available: Yes 2022-03-20T15:00:18.360 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ps 2022-03-20T15:00:18.944 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-20T15:00:18.944 INFO:teuthology.orchestra.run.smithi007.stdout:crashcollector.crash smithi007 running (109s) 0s ago 110s - - 7c0c85ada2e4 2022-03-20T15:00:18.945 INFO:teuthology.orchestra.run.smithi007.stdout:crashcollector.crash smithi057 running (3m) 0s ago 3m - - 7c0c85ada2e4 2022-03-20T15:00:18.945 INFO:teuthology.orchestra.run.smithi007.stdout:crashcollector.crash smithi087 running (3m) 0s ago 3m - - 7c0c85ada2e4 2022-03-20T15:00:18.945 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.a smithi007 running (3m) 0s ago 3m - - 7c0c85ada2e4 2022-03-20T15:00:18.946 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi087 running (6m) 0s ago 6m - - 7c0c85ada2e4 2022-03-20T15:00:18.946 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi057 running (5m) 0s ago 5m - - 7c0c85ada2e4 2022-03-20T15:00:18.946 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (4m) 0s ago 4m - - 7c0c85ada2e4 2022-03-20T15:00:18.946 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi057 running (113s) 0s ago 113s - - 7c0c85ada2e4 2022-03-20T15:00:18.947 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi057 running (113s) 0s ago 113s - - 7c0c85ada2e4 2022-03-20T15:00:18.947 INFO:teuthology.orchestra.run.smithi007.stdout:osd.10 smithi087 running (47s) 0s ago 47s - - 7c0c85ada2e4 2022-03-20T15:00:18.947 INFO:teuthology.orchestra.run.smithi007.stdout:osd.11 smithi087 running (50s) 0s ago 50s - - 7c0c85ada2e4 2022-03-20T15:00:18.947 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi057 running (112s) 0s ago 112s - - 7c0c85ada2e4 2022-03-20T15:00:18.947 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi057 running (112s) 0s ago 112s - - 7c0c85ada2e4 2022-03-20T15:00:18.948 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi087 running (112s) 0s ago 112s - - 7c0c85ada2e4 2022-03-20T15:00:18.948 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi007 running (110s) 0s ago 110s - - 7c0c85ada2e4 2022-03-20T15:00:18.948 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi087 running (103s) 0s ago 104s - - 7c0c85ada2e4 2022-03-20T15:00:18.948 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi007 running (107s) 0s ago 108s - - 7c0c85ada2e4 2022-03-20T15:00:18.949 INFO:teuthology.orchestra.run.smithi007.stdout:osd.8 smithi007 running (102s) 0s ago 102s - - 7c0c85ada2e4 2022-03-20T15:00:18.949 INFO:teuthology.orchestra.run.smithi007.stdout:osd.9 smithi007 running (99s) 0s ago 99s - - 7c0c85ada2e4 2022-03-20T15:00:18.965 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls 2022-03-20T15:00:19.660 INFO:teuthology.orchestra.run.smithi007.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-20T15:00:19.661 INFO:teuthology.orchestra.run.smithi007.stdout:crash 3/3 0s ago 3m * 2022-03-20T15:00:19.661 INFO:teuthology.orchestra.run.smithi007.stdout:mgr 1/1 0s ago 3m count:1 2022-03-20T15:00:19.661 INFO:teuthology.orchestra.run.smithi007.stdout:mon 3/3 0s ago 6m count:3 2022-03-20T15:00:19.661 INFO:teuthology.orchestra.run.smithi007.stdout:osd 24 0s ago 114s 2022-03-20T15:00:19.662 INFO:teuthology.orchestra.run.smithi007.stdout:osd.all-available-devices 0 0s ago - * 2022-03-20T15:00:19.687 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch device ls 2022-03-20T15:00:20.239 INFO:teuthology.orchestra.run.smithi007.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-20T15:00:20.240 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-20T15:00:20.240 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-20T15:00:20.240 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-20T15:00:20.241 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-20T15:00:20.241 INFO:teuthology.orchestra.run.smithi007.stdout:smithi057 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-20T15:00:20.241 INFO:teuthology.orchestra.run.smithi007.stdout:smithi057 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-20T15:00:20.242 INFO:teuthology.orchestra.run.smithi007.stdout:smithi057 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-20T15:00:20.242 INFO:teuthology.orchestra.run.smithi007.stdout:smithi057 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-20T15:00:20.242 INFO:teuthology.orchestra.run.smithi007.stdout:smithi087 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-20T15:00:20.242 INFO:teuthology.orchestra.run.smithi007.stdout:smithi087 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-20T15:00:20.243 INFO:teuthology.orchestra.run.smithi007.stdout:smithi087 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-20T15:00:20.243 INFO:teuthology.orchestra.run.smithi007.stdout:smithi087 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-20T15:00:20.263 INFO:teuthology.run_tasks:Running task exec... 2022-03-20T15:00:20.276 INFO:teuthology.task.exec:Executing custom commands... 2022-03-20T15:00:20.277 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi007.front.sepia.ceph.com 2022-03-20T15:00:20.277 DEBUG:teuthology.orchestra.run.smithi007:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'set -ex 2022-03-20T15:00:20.278 DEBUG:teuthology.orchestra.run.smithi007:> toolbox() { 2022-03-20T15:00:20.278 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- "$@" 2022-03-20T15:00:20.278 DEBUG:teuthology.orchestra.run.smithi007:> } 2022-03-20T15:00:20.279 DEBUG:teuthology.orchestra.run.smithi007:> orig_num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-20T15:00:20.279 DEBUG:teuthology.orchestra.run.smithi007:> toolbox ceph orch osd rm 0 --force 2022-03-20T15:00:20.279 DEBUG:teuthology.orchestra.run.smithi007:> removed_pv="" 2022-03-20T15:00:20.280 DEBUG:teuthology.orchestra.run.smithi007:> while [ "$removed_pv" = "" ] 2022-03-20T15:00:20.280 DEBUG:teuthology.orchestra.run.smithi007:> do 2022-03-20T15:00:20.280 DEBUG:teuthology.orchestra.run.smithi007:> removed_pv=`kubectl get pv | grep Released | cut -f1 -d " "` 2022-03-20T15:00:20.281 DEBUG:teuthology.orchestra.run.smithi007:> sleep 3s 2022-03-20T15:00:20.281 DEBUG:teuthology.orchestra.run.smithi007:> done 2022-03-20T15:00:20.281 DEBUG:teuthology.orchestra.run.smithi007:> target_path=`kubectl get pv $removed_pv -o jsonpath='"'"'{.spec.local.path}'"'"'` 2022-03-20T15:00:20.282 DEBUG:teuthology.orchestra.run.smithi007:> host=`echo $removed_pv | cut -f1 -d "-"` 2022-03-20T15:00:20.282 DEBUG:teuthology.orchestra.run.smithi007:> toolbox ceph orch device zap $host $target_path --force 2022-03-20T15:00:20.282 DEBUG:teuthology.orchestra.run.smithi007:> zap_completion="0" 2022-03-20T15:00:20.282 DEBUG:teuthology.orchestra.run.smithi007:> while [ "$zap_completion" = "0" ] 2022-03-20T15:00:20.283 DEBUG:teuthology.orchestra.run.smithi007:> do 2022-03-20T15:00:20.283 DEBUG:teuthology.orchestra.run.smithi007:> zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap -o jsonpath='"'"'{.status.succeeded.path}'"'"'` 2022-03-20T15:00:20.283 DEBUG:teuthology.orchestra.run.smithi007:> sleep 3s 2022-03-20T15:00:20.284 DEBUG:teuthology.orchestra.run.smithi007:> done 2022-03-20T15:00:20.284 DEBUG:teuthology.orchestra.run.smithi007:> kubectl patch pv $removed_pv -p '"'"'{"spec":{"claimRef": null}}'"'"' 2022-03-20T15:00:20.284 DEBUG:teuthology.orchestra.run.smithi007:> toolbox ceph orch apply osd --all-available-devices 2022-03-20T15:00:20.285 DEBUG:teuthology.orchestra.run.smithi007:> kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-20T15:00:20.285 DEBUG:teuthology.orchestra.run.smithi007:> num_osd="0" 2022-03-20T15:00:20.285 DEBUG:teuthology.orchestra.run.smithi007:> while [ "$num_osd" != "$orig_num_osd" ] 2022-03-20T15:00:20.286 DEBUG:teuthology.orchestra.run.smithi007:> do 2022-03-20T15:00:20.286 DEBUG:teuthology.orchestra.run.smithi007:> echo "waiting for osd to come back up" 2022-03-20T15:00:20.286 DEBUG:teuthology.orchestra.run.smithi007:> num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-20T15:00:20.287 DEBUG:teuthology.orchestra.run.smithi007:> sleep 30s 2022-03-20T15:00:20.287 DEBUG:teuthology.orchestra.run.smithi007:> done 2022-03-20T15:00:20.287 DEBUG:teuthology.orchestra.run.smithi007:> ' 2022-03-20T15:00:20.304 INFO:teuthology.orchestra.run.smithi007.stderr:++ toolbox ceph osd stat 2022-03-20T15:00:20.304 INFO:teuthology.orchestra.run.smithi007.stderr:++ cut -f3 -d ' ' 2022-03-20T15:00:20.305 INFO:teuthology.orchestra.run.smithi007.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-20T15:00:22.493 INFO:teuthology.orchestra.run.smithi007.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-20T15:00:22.971 INFO:teuthology.orchestra.run.smithi007.stderr:+ orig_num_osd=12 2022-03-20T15:00:22.971 INFO:teuthology.orchestra.run.smithi007.stderr:+ toolbox ceph orch osd rm 0 --force 2022-03-20T15:00:22.971 INFO:teuthology.orchestra.run.smithi007.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch osd rm 0 --force 2022-03-20T15:00:23.047 INFO:teuthology.orchestra.run.smithi007.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-20T15:00:23.661 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:23.655940 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{ 2022-03-20T15:00:23.662 INFO:tasks.rook.operator.smithi007.stdout:   CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", AllowUnsupported: true}, 2022-03-20T15:00:23.662 INFO:tasks.rook.operator.smithi007.stdout:   Storage: v1.StorageScopeSpec{ 2022-03-20T15:00:23.662 INFO:tasks.rook.operator.smithi007.stdout:   ... // 3 identical fields 2022-03-20T15:00:23.663 INFO:tasks.rook.operator.smithi007.stdout:   Config: nil, 2022-03-20T15:00:23.663 INFO:tasks.rook.operator.smithi007.stdout:   Selection: {}, 2022-03-20T15:00:23.663 INFO:tasks.rook.operator.smithi007.stdout:   StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-20T15:00:23.663 INFO:tasks.rook.operator.smithi007.stdout:   ... // 3 identical elements 2022-03-20T15:00:23.664 INFO:tasks.rook.operator.smithi007.stdout:   {Name: "smithi007-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi007-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-03-20T15:00:23.664 INFO:tasks.rook.operator.smithi007.stdout:   {Name: "smithi057-lv-1", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi057-lv-1", StorageClassName: &"scratch", ...}}}}, 2022-03-20T15:00:23.665 INFO:tasks.rook.operator.smithi007.stdout:-  { 2022-03-20T15:00:23.665 INFO:tasks.rook.operator.smithi007.stdout:-  Name: "smithi057-lv-2", 2022-03-20T15:00:23.665 INFO:tasks.rook.operator.smithi007.stdout:-  Count: 1, 2022-03-20T15:00:23.665 INFO:tasks.rook.operator.smithi007.stdout:-  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-20T15:00:23.666 INFO:tasks.rook.operator.smithi007.stdout:-  }, 2022-03-20T15:00:23.666 INFO:tasks.rook.operator.smithi007.stdout:   {Name: "smithi057-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi057-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-20T15:00:23.666 INFO:tasks.rook.operator.smithi007.stdout:   {Name: "smithi057-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi057-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-03-20T15:00:23.666 INFO:tasks.rook.operator.smithi007.stdout:   ... // 4 identical elements 2022-03-20T15:00:23.667 INFO:tasks.rook.operator.smithi007.stdout:   }, 2022-03-20T15:00:23.667 INFO:tasks.rook.operator.smithi007.stdout:   }, 2022-03-20T15:00:23.667 INFO:tasks.rook.operator.smithi007.stdout:   Annotations: nil, 2022-03-20T15:00:23.667 INFO:tasks.rook.operator.smithi007.stdout:   Labels: nil, 2022-03-20T15:00:23.668 INFO:tasks.rook.operator.smithi007.stdout:   ... // 20 identical fields 2022-03-20T15:00:23.668 INFO:tasks.rook.operator.smithi007.stdout:  } 2022-03-20T15:00:23.668 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:23.656078 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-20T15:00:23.669 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:23.656379 I | operator: successfully started the controller-runtime manager 2022-03-20T15:00:23.669 INFO:tasks.rook.operator.smithi007.stdout:I0320 15:00:23.656501 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-20T15:00:23.671 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:23.670262 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-20T15:00:23.672 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:23.670282 I | operator: watching all namespaces for Ceph CRs 2022-03-20T15:00:23.672 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:23.670307 I | operator: setting up schemes 2022-03-20T15:00:23.678 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:23.673708 I | operator: setting up the controller-runtime manager 2022-03-20T15:00:23.880 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:23.878830 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-20T15:00:23.926 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:23.924278 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-20T15:00:23.927 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:23.924313 I | ceph-cluster-controller: successfully started 2022-03-20T15:00:23.927 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:23.924355 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-20T15:00:23.927 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:23.924369 I | ceph-crashcollector-controller: successfully started 2022-03-20T15:00:23.927 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:23.924386 I | ceph-block-pool-controller: successfully started 2022-03-20T15:00:23.928 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:23.924401 I | ceph-object-store-user-controller: successfully started 2022-03-20T15:00:23.928 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:23.924414 I | ceph-object-realm-controller: successfully started 2022-03-20T15:00:23.928 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:23.924426 I | ceph-object-zonegroup-controller: successfully started 2022-03-20T15:00:23.928 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:23.924434 I | ceph-object-zone-controller: successfully started 2022-03-20T15:00:23.929 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:23.924570 I | ceph-object-controller: successfully started 2022-03-20T15:00:23.929 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:23.924600 I | ceph-file-controller: successfully started 2022-03-20T15:00:23.929 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:23.924625 I | ceph-nfs-controller: successfully started 2022-03-20T15:00:23.929 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:23.924647 I | ceph-rbd-mirror-controller: successfully started 2022-03-20T15:00:23.930 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:23.924670 I | ceph-client-controller: successfully started 2022-03-20T15:00:23.931 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:23.924681 I | ceph-filesystem-mirror-controller: successfully started 2022-03-20T15:00:23.931 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:23.924701 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-20T15:00:23.931 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:23.924723 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-20T15:00:23.931 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:23.924740 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-20T15:00:23.932 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:23.924754 I | ceph-bucket-topic: successfully started 2022-03-20T15:00:23.932 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:23.924764 I | ceph-bucket-notification: successfully started 2022-03-20T15:00:23.932 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:23.924775 I | ceph-bucket-notification: successfully started 2022-03-20T15:00:23.932 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:23.924788 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-20T15:00:23.933 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:23.926628 I | operator: starting the controller-runtime manager 2022-03-20T15:00:24.035 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:24.033542 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-20T15:00:24.036 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:24.033777 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-20T15:00:24.036 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:24.033835 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-20T15:00:24.036 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:24.033868 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-20T15:00:24.112 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:24.110557 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-20T15:00:24.143 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:24.141026 I | op-mon: parsing mon endpoints: a=10.98.115.144:6789,b=10.109.19.50:6789,c=10.106.65.43:6789 2022-03-20T15:00:24.143 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:24.141150 I | op-mon: parsing mon endpoints: a=10.98.115.144:6789,b=10.109.19.50:6789,c=10.106.65.43:6789 2022-03-20T15:00:24.144 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:24.141245 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-20T15:00:24.144 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:24.141273 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-20T15:00:24.145 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:24.143246 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-20T15:00:24.146 INFO:tasks.rook.operator.smithi007.stdout:I0320 15:00:24.143324 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-20T15:00:24.206 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:24.204392 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-20T15:00:24.319 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:24.316661 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-20T15:00:24.320 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:24.316694 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-20T15:00:24.320 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:24.316703 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-20T15:00:24.320 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:24.316711 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-20T15:00:24.320 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:24.316721 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-20T15:00:24.321 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:24.316729 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-20T15:00:24.321 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:24.316739 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-20T15:00:24.321 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:24.316748 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-20T15:00:24.322 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:24.316757 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-20T15:00:24.322 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:24.316769 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-20T15:00:24.322 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:24.316785 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-20T15:00:24.322 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:24.316800 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-20T15:00:24.323 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:24.316814 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-20T15:00:24.323 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:24.316821 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-20T15:00:24.323 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:24.316828 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-20T15:00:24.323 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:24.316849 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-20T15:00:24.324 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:24.316945 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-20T15:00:24.324 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:24.316969 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-20T15:00:24.328 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:24.320801 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e... 2022-03-20T15:00:29.250 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:29.249163 I | ceph-spec: object "rook-ceph-osd-0" matched on delete, reconciling 2022-03-20T15:00:29.977 INFO:teuthology.orchestra.run.smithi007.stdout:Success 2022-03-20T15:00:30.001 INFO:teuthology.orchestra.run.smithi007.stderr:+ removed_pv= 2022-03-20T15:00:30.002 INFO:teuthology.orchestra.run.smithi007.stderr:+ '[' '' = '' ']' 2022-03-20T15:00:30.003 INFO:teuthology.orchestra.run.smithi007.stderr:++ kubectl get pv 2022-03-20T15:00:30.003 INFO:teuthology.orchestra.run.smithi007.stderr:++ grep Released 2022-03-20T15:00:30.004 INFO:teuthology.orchestra.run.smithi007.stderr:++ cut -f1 -d ' ' 2022-03-20T15:00:30.079 INFO:teuthology.orchestra.run.smithi007.stderr:+ removed_pv=smithi057-lv-2 2022-03-20T15:00:30.080 INFO:teuthology.orchestra.run.smithi007.stderr:+ sleep 3s 2022-03-20T15:00:33.080 INFO:teuthology.orchestra.run.smithi007.stderr:+ '[' smithi057-lv-2 = '' ']' 2022-03-20T15:00:33.081 INFO:teuthology.orchestra.run.smithi007.stderr:++ kubectl get pv smithi057-lv-2 -o 'jsonpath={.spec.local.path}' 2022-03-20T15:00:33.149 INFO:teuthology.orchestra.run.smithi007.stderr:+ target_path=/dev/vg_nvme/lv_2 2022-03-20T15:00:33.150 INFO:teuthology.orchestra.run.smithi007.stderr:++ echo smithi057-lv-2 2022-03-20T15:00:33.150 INFO:teuthology.orchestra.run.smithi007.stderr:++ cut -f1 -d - 2022-03-20T15:00:33.151 INFO:teuthology.orchestra.run.smithi007.stderr:+ host=smithi057 2022-03-20T15:00:33.152 INFO:teuthology.orchestra.run.smithi007.stderr:+ toolbox ceph orch device zap smithi057 /dev/vg_nvme/lv_2 --force 2022-03-20T15:00:33.152 INFO:teuthology.orchestra.run.smithi007.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch device zap smithi057 /dev/vg_nvme/lv_2 --force 2022-03-20T15:00:33.235 INFO:teuthology.orchestra.run.smithi007.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-20T15:00:33.716 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/vg_nvme/lv_2 on smithi057 zapped 2022-03-20T15:00:33.741 INFO:teuthology.orchestra.run.smithi007.stderr:+ zap_completion=0 2022-03-20T15:00:33.741 INFO:teuthology.orchestra.run.smithi007.stderr:+ '[' 0 = 0 ']' 2022-03-20T15:00:33.741 INFO:teuthology.orchestra.run.smithi007.stderr:++ kubectl get job -n rook-ceph rook-ceph-device-zap -o 'jsonpath={.status.succeeded.path}' 2022-03-20T15:00:33.866 INFO:teuthology.orchestra.run.smithi007.stderr:+ zap_completion= 2022-03-20T15:00:33.867 INFO:teuthology.orchestra.run.smithi007.stderr:+ sleep 3s 2022-03-20T15:00:34.184 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.180702 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-20T15:00:34.184 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.180735 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-20T15:00:34.184 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.180744 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-20T15:00:34.184 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.180748 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-20T15:00:34.185 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.180753 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-20T15:00:34.185 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.180757 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-20T15:00:34.186 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.180762 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-20T15:00:34.186 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.180766 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-20T15:00:34.186 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.180770 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-20T15:00:34.186 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.180774 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-20T15:00:34.187 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.180778 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-20T15:00:34.187 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.180782 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-20T15:00:34.187 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.180788 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-20T15:00:34.187 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.180792 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-20T15:00:34.188 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.180797 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-20T15:00:34.188 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.180801 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-20T15:00:34.188 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.180807 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-20T15:00:34.188 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.180811 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-20T15:00:34.189 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.180815 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-20T15:00:34.189 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.180820 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-20T15:00:34.189 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.180824 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-20T15:00:34.189 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.180827 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-20T15:00:34.190 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.180830 I | ceph-csi: Kubernetes version is 1.23 2022-03-20T15:00:34.190 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.180834 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-20T15:00:34.190 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.180837 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-20T15:00:34.191 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.184759 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-20T15:00:34.193 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.189082 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-20T15:00:34.194 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.189098 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-20T15:00:34.194 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.189103 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-20T15:00:34.194 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.189107 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-20T15:00:34.195 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.189111 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-20T15:00:34.195 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.189115 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-20T15:00:34.195 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.189119 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-20T15:00:34.258 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.256084 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-20T15:00:34.258 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.256102 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-20T15:00:34.258 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.256106 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-20T15:00:34.341 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.340061 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-20T15:00:34.601 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.599844 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-20T15:00:34.602 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.599877 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-20T15:00:34.602 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.599891 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-20T15:00:34.605 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.601030 I | ceph-spec: detected ceph image version: "17.0.0-11187 quincy" 2022-03-20T15:00:34.605 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.601057 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-20T15:00:34.751 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.749844 I | op-mon: parsing mon endpoints: a=10.98.115.144:6789,b=10.109.19.50:6789,c=10.106.65.43:6789 2022-03-20T15:00:34.798 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.796835 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-20T15:00:34.798 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.796853 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-20T15:00:34.799 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.796859 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-20T15:00:34.842 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.840519 I | ceph-csi: successfully started CSI CephFS driver 2022-03-20T15:00:34.970 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.969007 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T15:00:34.973 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:34.969176 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T15:00:35.043 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:35.041917 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-20T15:00:35.047 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:35.046149 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-20T15:00:35.047 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:35.046173 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-20T15:00:35.053 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:35.050272 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-20T15:00:35.622 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:35.620801 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11187 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e" 2022-03-20T15:00:35.698 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:35.696486 I | op-mon: start running mons 2022-03-20T15:00:35.720 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:35.718821 I | op-mon: parsing mon endpoints: a=10.98.115.144:6789,b=10.109.19.50:6789,c=10.106.65.43:6789 2022-03-20T15:00:36.467 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:36.465708 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.98.115.144:6789","10.109.19.50:6789","10.106.65.43:6789"]}] data:a=10.98.115.144:6789,b=10.109.19.50:6789,c=10.106.65.43:6789 mapping:{"node":{"a":{"Name":"smithi087","Hostname":"smithi087","Address":"172.21.15.87"},"b":{"Name":"smithi057","Hostname":"smithi057","Address":"172.21.15.57"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:2] 2022-03-20T15:00:36.868 INFO:teuthology.orchestra.run.smithi007.stderr:+ '[' '' = 0 ']' 2022-03-20T15:00:36.868 INFO:teuthology.orchestra.run.smithi007.stderr:+ kubectl patch pv smithi057-lv-2 -p '{"spec":{"claimRef": null}}' 2022-03-20T15:00:36.954 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi057-lv-2 patched 2022-03-20T15:00:36.956 INFO:teuthology.orchestra.run.smithi007.stderr:+ toolbox ceph orch apply osd --all-available-devices 2022-03-20T15:00:36.957 INFO:teuthology.orchestra.run.smithi007.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch apply osd --all-available-devices 2022-03-20T15:00:37.032 INFO:teuthology.orchestra.run.smithi007.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-20T15:00:37.066 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:37.065000 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T15:00:37.071 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:37.065330 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T15:00:37.909 INFO:teuthology.orchestra.run.smithi007.stdout:['Success'] 2022-03-20T15:00:37.916 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:37.914201 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{ 2022-03-20T15:00:37.916 INFO:tasks.rook.operator.smithi007.stdout:   CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", AllowUnsupported: true}, 2022-03-20T15:00:37.917 INFO:tasks.rook.operator.smithi007.stdout:   Storage: v1.StorageScopeSpec{ 2022-03-20T15:00:37.917 INFO:tasks.rook.operator.smithi007.stdout:   ... // 3 identical fields 2022-03-20T15:00:37.917 INFO:tasks.rook.operator.smithi007.stdout:   Config: nil, 2022-03-20T15:00:37.917 INFO:tasks.rook.operator.smithi007.stdout:   Selection: {}, 2022-03-20T15:00:37.918 INFO:tasks.rook.operator.smithi007.stdout:   StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-20T15:00:37.919 INFO:tasks.rook.operator.smithi007.stdout:   ... // 9 identical elements 2022-03-20T15:00:37.919 INFO:tasks.rook.operator.smithi007.stdout:   {Name: "smithi087-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi087-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-20T15:00:37.920 INFO:tasks.rook.operator.smithi007.stdout:   {Name: "smithi087-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi087-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-03-20T15:00:37.920 INFO:tasks.rook.operator.smithi007.stdout:+  { 2022-03-20T15:00:37.920 INFO:tasks.rook.operator.smithi007.stdout:+  Name: "smithi057-lv-2", 2022-03-20T15:00:37.921 INFO:tasks.rook.operator.smithi007.stdout:+  Count: 1, 2022-03-20T15:00:37.921 INFO:tasks.rook.operator.smithi007.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-20T15:00:37.921 INFO:tasks.rook.operator.smithi007.stdout:+  }, 2022-03-20T15:00:37.922 INFO:tasks.rook.operator.smithi007.stdout:   }, 2022-03-20T15:00:37.922 INFO:tasks.rook.operator.smithi007.stdout:   }, 2022-03-20T15:00:37.922 INFO:tasks.rook.operator.smithi007.stdout:   Annotations: nil, 2022-03-20T15:00:37.923 INFO:tasks.rook.operator.smithi007.stdout:   Labels: nil, 2022-03-20T15:00:37.923 INFO:tasks.rook.operator.smithi007.stdout:   ... // 20 identical fields 2022-03-20T15:00:37.924 INFO:tasks.rook.operator.smithi007.stdout:  } 2022-03-20T15:00:37.924 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:37.914304 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-20T15:00:37.924 INFO:tasks.rook.operator.smithi007.stdout:I0320 15:00:37.914737 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-20T15:00:37.927 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:37.925689 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph monitors: failed to initialize ceph cluster info: failed to save admin keyring secret: failed to get secret for rook-ceph-admin-keyring: context canceled 2022-03-20T15:00:37.927 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:37.925802 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-20T15:00:37.928 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:37.925906 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to load cluster info: failed to get mon secrets: context canceled 2022-03-20T15:00:37.928 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:37.925946 I | operator: successfully started the controller-runtime manager 2022-03-20T15:00:37.930 INFO:teuthology.orchestra.run.smithi007.stderr:+ kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-20T15:00:37.998 INFO:teuthology.orchestra.run.smithi007.stdout:job.batch "rook-ceph-device-zap" deleted 2022-03-20T15:00:38.029 INFO:teuthology.orchestra.run.smithi007.stderr:+ num_osd=0 2022-03-20T15:00:38.029 INFO:teuthology.orchestra.run.smithi007.stderr:+ '[' 0 '!=' 12 ']' 2022-03-20T15:00:38.029 INFO:teuthology.orchestra.run.smithi007.stderr:+ echo 'waiting for osd to come back up' 2022-03-20T15:00:38.030 INFO:teuthology.orchestra.run.smithi007.stdout:waiting for osd to come back up 2022-03-20T15:00:38.030 INFO:teuthology.orchestra.run.smithi007.stderr:++ toolbox ceph osd stat 2022-03-20T15:00:38.031 INFO:teuthology.orchestra.run.smithi007.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-20T15:00:38.031 INFO:teuthology.orchestra.run.smithi007.stderr:++ cut -f3 -d ' ' 2022-03-20T15:00:38.068 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:38.066341 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-20T15:00:38.068 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:38.066373 I | operator: watching all namespaces for Ceph CRs 2022-03-20T15:00:38.069 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:38.066413 I | operator: setting up schemes 2022-03-20T15:00:38.072 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:38.070070 I | operator: setting up the controller-runtime manager 2022-03-20T15:00:38.130 INFO:teuthology.orchestra.run.smithi007.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-20T15:00:38.275 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:38.273422 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-20T15:00:38.280 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:38.276197 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-20T15:00:38.280 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:38.276245 I | ceph-cluster-controller: successfully started 2022-03-20T15:00:38.280 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:38.276314 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-20T15:00:38.281 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:38.276355 I | ceph-crashcollector-controller: successfully started 2022-03-20T15:00:38.281 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:38.276389 I | ceph-block-pool-controller: successfully started 2022-03-20T15:00:38.281 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:38.276423 I | ceph-object-store-user-controller: successfully started 2022-03-20T15:00:38.281 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:38.276462 I | ceph-object-realm-controller: successfully started 2022-03-20T15:00:38.282 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:38.276482 I | ceph-object-zonegroup-controller: successfully started 2022-03-20T15:00:38.282 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:38.276502 I | ceph-object-zone-controller: successfully started 2022-03-20T15:00:38.282 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:38.276707 I | ceph-object-controller: successfully started 2022-03-20T15:00:38.282 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:38.276762 I | ceph-file-controller: successfully started 2022-03-20T15:00:38.283 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:38.276822 I | ceph-nfs-controller: successfully started 2022-03-20T15:00:38.283 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:38.276860 I | ceph-rbd-mirror-controller: successfully started 2022-03-20T15:00:38.283 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:38.276897 I | ceph-client-controller: successfully started 2022-03-20T15:00:38.283 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:38.276920 I | ceph-filesystem-mirror-controller: successfully started 2022-03-20T15:00:38.284 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:38.276965 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-20T15:00:38.284 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:38.276988 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-20T15:00:38.284 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:38.277007 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-20T15:00:38.284 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:38.277039 I | ceph-bucket-topic: successfully started 2022-03-20T15:00:38.285 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:38.277056 I | ceph-bucket-notification: successfully started 2022-03-20T15:00:38.285 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:38.277075 I | ceph-bucket-notification: successfully started 2022-03-20T15:00:38.285 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:38.277091 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-20T15:00:38.286 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:38.278385 I | operator: starting the controller-runtime manager 2022-03-20T15:00:38.395 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:38.383593 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-20T15:00:38.396 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:38.383665 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-20T15:00:38.396 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:38.383695 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-20T15:00:38.397 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:38.383705 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-20T15:00:38.397 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:38.387255 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-20T15:00:38.624 INFO:teuthology.orchestra.run.smithi007.stderr:+ num_osd=11 2022-03-20T15:00:38.624 INFO:teuthology.orchestra.run.smithi007.stderr:+ sleep 30s 2022-03-20T15:00:39.064 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:39.063014 I | op-mon: parsing mon endpoints: a=10.98.115.144:6789,b=10.109.19.50:6789,c=10.106.65.43:6789 2022-03-20T15:00:39.086 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:39.084995 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e... 2022-03-20T15:00:39.297 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:39.295703 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-20T15:00:39.466 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:39.464718 I | op-mon: parsing mon endpoints: a=10.98.115.144:6789,b=10.109.19.50:6789,c=10.106.65.43:6789 2022-03-20T15:00:39.467 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:39.464808 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-20T15:00:39.467 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:39.464828 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-20T15:00:39.471 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:39.467345 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-20T15:00:39.472 INFO:tasks.rook.operator.smithi007.stdout:I0320 15:00:39.467456 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-20T15:00:39.889 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:39.887232 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-20T15:00:39.889 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:39.887252 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-20T15:00:39.889 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:39.887255 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-20T15:00:39.890 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:39.887259 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-20T15:00:39.890 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:39.887264 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-20T15:00:39.891 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:39.887269 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-20T15:00:39.891 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:39.887275 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-20T15:00:39.891 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:39.887281 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-20T15:00:39.892 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:39.887286 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-20T15:00:39.892 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:39.887291 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-20T15:00:39.892 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:39.887296 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-20T15:00:39.892 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:39.887302 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-20T15:00:39.893 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:39.887307 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-20T15:00:39.893 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:39.887312 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-20T15:00:39.893 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:39.887316 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-20T15:00:39.893 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:39.887322 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-20T15:00:39.893 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:39.887379 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-20T15:00:39.894 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:39.887395 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-20T15:00:39.894 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:39.890554 I | op-k8sutil: Removing previous job rook-ceph-detect-version to start a new one 2022-03-20T15:00:39.973 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:39.971468 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-20T15:00:42.977 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:42.975377 I | op-k8sutil: batch job rook-ceph-detect-version deleted 2022-03-20T15:00:45.982 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:45.978164 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-20T15:00:45.982 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:45.978201 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-20T15:00:45.983 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:45.978213 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-20T15:00:45.983 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:45.978220 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-20T15:00:45.983 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:45.978228 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-20T15:00:45.983 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:45.978236 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-20T15:00:45.984 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:45.978248 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-20T15:00:45.984 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:45.978269 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-20T15:00:45.984 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:45.978296 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-20T15:00:45.984 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:45.978307 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-20T15:00:45.985 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:45.978323 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-20T15:00:45.985 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:45.978336 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-20T15:00:45.985 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:45.978342 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-20T15:00:45.985 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:45.978350 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-20T15:00:45.986 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:45.978356 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-20T15:00:45.986 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:45.978366 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-20T15:00:45.986 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:45.978372 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-20T15:00:45.987 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:45.978381 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-20T15:00:45.987 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:45.978392 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-20T15:00:45.987 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:45.978405 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-20T15:00:45.988 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:45.978412 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-20T15:00:45.988 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:45.978423 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-20T15:00:45.988 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:45.978429 I | ceph-csi: Kubernetes version is 1.23 2022-03-20T15:00:45.989 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:45.978440 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-20T15:00:45.989 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:45.978447 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-20T15:00:45.990 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:45.983492 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-20T15:00:45.992 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:45.989825 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-20T15:00:45.992 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:45.989844 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-20T15:00:45.992 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:45.989848 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-20T15:00:45.993 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:45.989852 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-20T15:00:45.993 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:45.989856 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-20T15:00:45.993 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:45.989860 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-20T15:00:45.993 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:45.989864 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-20T15:00:46.061 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:46.059283 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-20T15:00:46.061 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:46.059312 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-20T15:00:46.062 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:46.059321 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-20T15:00:46.118 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:46.113398 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-20T15:00:46.310 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:46.308440 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-20T15:00:46.310 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:46.308483 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-20T15:00:46.311 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:46.308499 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-20T15:00:46.355 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:46.353338 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-20T15:00:46.355 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:46.353364 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-20T15:00:46.355 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:46.353372 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-20T15:00:46.432 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:46.430427 I | ceph-csi: successfully started CSI CephFS driver 2022-03-20T15:00:46.655 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:46.652844 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-20T15:00:46.674 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:46.672791 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-20T15:00:46.675 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:46.672809 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-20T15:00:46.686 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:46.684706 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-20T15:00:51.831 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:51.829267 I | ceph-spec: detected ceph image version: "17.0.0-11187 quincy" 2022-03-20T15:00:51.832 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:51.829304 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-20T15:00:51.896 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:51.894946 I | op-mon: parsing mon endpoints: a=10.98.115.144:6789,b=10.109.19.50:6789,c=10.106.65.43:6789 2022-03-20T15:00:52.001 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:51.999841 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T15:00:52.062 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:52.060248 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T15:00:52.496 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:52.494651 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11187 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e" 2022-03-20T15:00:52.610 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:52.608892 I | op-mon: start running mons 2022-03-20T15:00:52.666 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:52.664525 I | op-mon: parsing mon endpoints: a=10.98.115.144:6789,b=10.109.19.50:6789,c=10.106.65.43:6789 2022-03-20T15:00:52.740 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:52.738591 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.98.115.144:6789","10.109.19.50:6789","10.106.65.43:6789"]}] data:a=10.98.115.144:6789,b=10.109.19.50:6789,c=10.106.65.43:6789 mapping:{"node":{"a":{"Name":"smithi087","Hostname":"smithi087","Address":"172.21.15.87"},"b":{"Name":"smithi057","Hostname":"smithi057","Address":"172.21.15.57"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:2] 2022-03-20T15:00:52.947 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:52.945694 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T15:00:52.951 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:52.946131 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T15:00:54.546 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:54.544873 I | op-mon: targeting the mon count 3 2022-03-20T15:00:54.558 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:54.556907 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T15:00:54.924 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:54.922596 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-20T15:00:54.925 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:54.922622 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T15:00:55.279 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:55.277123 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-20T15:00:55.279 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:55.277145 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T15:00:55.631 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:55.629132 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-20T15:00:55.631 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:55.629164 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-20T15:00:55.969 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:55.967157 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-20T15:00:55.969 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:55.967197 I | op-config: deleting "log file" option from the mon configuration database 2022-03-20T15:00:56.353 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:56.351063 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-20T15:00:56.353 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:56.351087 I | op-mon: checking for basic quorum with existing mons 2022-03-20T15:00:56.475 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:56.473892 I | op-mon: mon "a" ip is 10.98.115.144 2022-03-20T15:00:56.566 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:56.564588 I | op-mon: mon "b" ip is 10.109.19.50 2022-03-20T15:00:56.950 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:56.948571 I | op-mon: mon "c" ip is 10.106.65.43 2022-03-20T15:00:57.549 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:57.547535 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.109.19.50:6789","10.106.65.43:6789","10.98.115.144:6789"]}] data:b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 mapping:{"node":{"a":{"Name":"smithi087","Hostname":"smithi087","Address":"172.21.15.87"},"b":{"Name":"smithi057","Hostname":"smithi057","Address":"172.21.15.57"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:2] 2022-03-20T15:00:58.145 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:58.142885 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-20T15:00:58.148 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:58.143194 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-20T15:00:58.574 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:58.572168 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-20T15:00:58.583 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:58.582177 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-20T15:00:58.584 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:58.582207 I | op-mon: waiting for mon quorum with [a b c] 2022-03-20T15:00:59.155 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:59.153261 I | op-mon: mons running: [a b c] 2022-03-20T15:00:59.571 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:59.569926 I | op-mon: Monitors in quorum: [a b c] 2022-03-20T15:00:59.578 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:59.576712 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-20T15:00:59.589 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:59.587268 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-20T15:00:59.589 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:59.587298 I | op-mon: waiting for mon quorum with [a b c] 2022-03-20T15:00:59.754 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:00:59.752739 I | op-mon: mons running: [a b c] 2022-03-20T15:01:00.175 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:00.173033 I | op-mon: Monitors in quorum: [a b c] 2022-03-20T15:01:00.183 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:00.181541 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-20T15:01:00.190 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:00.188529 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-20T15:01:00.190 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:00.188547 I | op-mon: waiting for mon quorum with [a b c] 2022-03-20T15:01:00.355 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:00.353512 I | op-mon: mons running: [a b c] 2022-03-20T15:01:00.766 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:00.764342 I | op-mon: Monitors in quorum: [a b c] 2022-03-20T15:01:00.766 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:00.764374 I | op-mon: mons created: 3 2022-03-20T15:01:01.171 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:01.169035 I | op-mon: waiting for mon quorum with [a b c] 2022-03-20T15:01:01.211 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:01.209986 I | op-mon: mons running: [a b c] 2022-03-20T15:01:01.617 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:01.615549 I | op-mon: Monitors in quorum: [a b c] 2022-03-20T15:01:01.617 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:01.615589 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-20T15:01:02.078 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:02.076926 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-20T15:01:02.525 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:02.523074 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-20T15:01:02.939 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:02.937349 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-20T15:01:03.442 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:03.440040 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-20T15:01:03.442 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:03.440065 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-20T15:01:03.870 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:03.867933 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-20T15:01:03.870 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:03.867968 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-20T15:01:03.870 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:03.867978 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-20T15:01:04.386 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:04.384432 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-20T15:01:04.415 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:04.413552 I | op-mgr: start running mgr 2022-03-20T15:01:04.415 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:04.413593 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-20T15:01:04.892 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:04.890862 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-20T15:01:04.903 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:04.901143 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-20T15:01:04.906 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:04.904553 I | op-mgr: setting services to point to mgr "a" 2022-03-20T15:01:05.423 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.421641 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-20T15:01:05.424 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.421825 I | op-mgr: successful modules: balancer 2022-03-20T15:01:05.438 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.436136 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-20T15:01:05.438 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.436172 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-20T15:01:05.498 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.496482 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-20T15:01:05.628 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.623576 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi007-lv-1" 2022-03-20T15:01:05.628 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.623604 I | op-osd: OSD PVC "smithi007-lv-1-data-0fltgg" already exists 2022-03-20T15:01:05.629 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.623613 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi007-lv-2" 2022-03-20T15:01:05.629 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.623628 I | op-osd: OSD PVC "smithi007-lv-2-data-0z6lbn" already exists 2022-03-20T15:01:05.629 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.623634 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi007-lv-3" 2022-03-20T15:01:05.629 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.623640 I | op-osd: OSD PVC "smithi007-lv-3-data-0wb6kn" already exists 2022-03-20T15:01:05.630 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.623644 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi007-lv-4" 2022-03-20T15:01:05.630 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.623649 I | op-osd: OSD PVC "smithi007-lv-4-data-0snpfz" already exists 2022-03-20T15:01:05.631 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.623652 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi057-lv-1" 2022-03-20T15:01:05.631 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.623656 I | op-osd: OSD PVC "smithi057-lv-1-data-056qwk" already exists 2022-03-20T15:01:05.631 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.623660 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi057-lv-3" 2022-03-20T15:01:05.631 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.623665 I | op-osd: OSD PVC "smithi057-lv-3-data-0vrztn" already exists 2022-03-20T15:01:05.632 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.623668 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi057-lv-4" 2022-03-20T15:01:05.632 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.623674 I | op-osd: OSD PVC "smithi057-lv-4-data-0zt8b4" already exists 2022-03-20T15:01:05.632 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.623685 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi087-lv-1" 2022-03-20T15:01:05.632 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.623694 I | op-osd: OSD PVC "smithi087-lv-1-data-0p7bb2" already exists 2022-03-20T15:01:05.633 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.623700 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi087-lv-2" 2022-03-20T15:01:05.633 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.623709 I | op-osd: OSD PVC "smithi087-lv-2-data-04lndq" already exists 2022-03-20T15:01:05.633 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.623718 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi087-lv-3" 2022-03-20T15:01:05.633 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.623728 I | op-osd: OSD PVC "smithi087-lv-3-data-0brh6f" already exists 2022-03-20T15:01:05.633 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.623740 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi087-lv-4" 2022-03-20T15:01:05.634 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.623747 I | op-osd: OSD PVC "smithi087-lv-4-data-0vssdn" already exists 2022-03-20T15:01:05.634 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.623751 I | op-osd: creating 1 new PVCs for device set "smithi057-lv-2" 2022-03-20T15:01:05.827 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.825930 I | op-osd: successfully provisioned PVC "smithi057-lv-2-data-04c4hs" 2022-03-20T15:01:05.872 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.865084 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-1-data-0fltgg" 2022-03-20T15:01:05.872 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.865101 I | op-osd: skipping OSD prepare job creation for PVC "smithi007-lv-1-data-0fltgg" because OSD daemon using the PVC already exists 2022-03-20T15:01:05.873 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.865105 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-2-data-0z6lbn" 2022-03-20T15:01:05.873 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.865108 I | op-osd: skipping OSD prepare job creation for PVC "smithi007-lv-2-data-0z6lbn" because OSD daemon using the PVC already exists 2022-03-20T15:01:05.874 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.865111 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-3-data-0wb6kn" 2022-03-20T15:01:05.874 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.865113 I | op-osd: skipping OSD prepare job creation for PVC "smithi007-lv-3-data-0wb6kn" because OSD daemon using the PVC already exists 2022-03-20T15:01:05.874 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.865116 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-4-data-0snpfz" 2022-03-20T15:01:05.875 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.865119 I | op-osd: skipping OSD prepare job creation for PVC "smithi007-lv-4-data-0snpfz" because OSD daemon using the PVC already exists 2022-03-20T15:01:05.875 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.865122 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-1-data-056qwk" 2022-03-20T15:01:05.875 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.865124 I | op-osd: skipping OSD prepare job creation for PVC "smithi057-lv-1-data-056qwk" because OSD daemon using the PVC already exists 2022-03-20T15:01:05.876 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.865128 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-3-data-0vrztn" 2022-03-20T15:01:05.876 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.865132 I | op-osd: skipping OSD prepare job creation for PVC "smithi057-lv-3-data-0vrztn" because OSD daemon using the PVC already exists 2022-03-20T15:01:05.876 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.865136 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-4-data-0zt8b4" 2022-03-20T15:01:05.877 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.865140 I | op-osd: skipping OSD prepare job creation for PVC "smithi057-lv-4-data-0zt8b4" because OSD daemon using the PVC already exists 2022-03-20T15:01:05.877 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.865143 I | op-osd: OSD will have its main bluestore block on "smithi087-lv-1-data-0p7bb2" 2022-03-20T15:01:05.877 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.865145 I | op-osd: skipping OSD prepare job creation for PVC "smithi087-lv-1-data-0p7bb2" because OSD daemon using the PVC already exists 2022-03-20T15:01:05.877 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.865148 I | op-osd: OSD will have its main bluestore block on "smithi087-lv-2-data-04lndq" 2022-03-20T15:01:05.878 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.865150 I | op-osd: skipping OSD prepare job creation for PVC "smithi087-lv-2-data-04lndq" because OSD daemon using the PVC already exists 2022-03-20T15:01:05.878 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.865153 I | op-osd: OSD will have its main bluestore block on "smithi087-lv-3-data-0brh6f" 2022-03-20T15:01:05.878 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.865157 I | op-osd: skipping OSD prepare job creation for PVC "smithi087-lv-3-data-0brh6f" because OSD daemon using the PVC already exists 2022-03-20T15:01:05.879 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.865160 I | op-osd: OSD will have its main bluestore block on "smithi087-lv-4-data-0vssdn" 2022-03-20T15:01:05.879 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.865163 I | op-osd: skipping OSD prepare job creation for PVC "smithi087-lv-4-data-0vssdn" because OSD daemon using the PVC already exists 2022-03-20T15:01:05.879 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:05.865166 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-2-data-04c4hs" 2022-03-20T15:01:06.248 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:06.246184 I | op-osd: started OSD provisioning job for PVC "smithi057-lv-2-data-04c4hs" 2022-03-20T15:01:06.248 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:06.246216 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-20T15:01:06.249 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:06.246224 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-20T15:01:06.425 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:06.423213 I | op-osd: OSD orchestration status for PVC smithi057-lv-2-data-04c4hs is "starting" 2022-03-20T15:01:06.526 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:06.524636 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:01:06.533 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:06.531011 I | op-osd: updating OSD 1 on PVC "smithi057-lv-4-data-0zt8b4" 2022-03-20T15:01:06.533 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:06.531048 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-4-data-0zt8b4" 2022-03-20T15:01:06.640 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:06.638361 I | op-mgr: successful modules: dashboard 2022-03-20T15:01:06.643 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:06.641583 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-20T15:01:06.654 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:06.653004 I | op-mgr: successful modules: prometheus 2022-03-20T15:01:06.685 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:06.683208 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:01:06.710 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:06.708564 I | op-osd: updating OSD 10 on PVC "smithi087-lv-3-data-0brh6f" 2022-03-20T15:01:06.711 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:06.708599 I | op-osd: OSD will have its main bluestore block on "smithi087-lv-3-data-0brh6f" 2022-03-20T15:01:06.945 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:06.940389 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:01:06.950 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:06.949064 I | op-osd: updating OSD 11 on PVC "smithi087-lv-2-data-04lndq" 2022-03-20T15:01:06.951 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:06.949088 I | op-osd: OSD will have its main bluestore block on "smithi087-lv-2-data-04lndq" 2022-03-20T15:01:07.113 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:07.111441 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:01:07.133 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:07.132158 I | op-osd: updating OSD 2 on PVC "smithi057-lv-1-data-056qwk" 2022-03-20T15:01:07.134 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:07.132194 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-1-data-056qwk" 2022-03-20T15:01:07.838 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:07.836673 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:01:07.934 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:07.932650 I | op-osd: updating OSD 3 on PVC "smithi057-lv-3-data-0vrztn" 2022-03-20T15:01:07.935 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:07.932691 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-3-data-0vrztn" 2022-03-20T15:01:08.624 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:08.622071 I | op-mgr: successful modules: orchestrator modules 2022-03-20T15:01:08.625 INFO:teuthology.orchestra.run.smithi007.stderr:+ '[' 11 '!=' 12 ']' 2022-03-20T15:01:08.626 INFO:teuthology.orchestra.run.smithi007.stderr:+ echo 'waiting for osd to come back up' 2022-03-20T15:01:08.626 INFO:teuthology.orchestra.run.smithi007.stdout:waiting for osd to come back up 2022-03-20T15:01:08.627 INFO:teuthology.orchestra.run.smithi007.stderr:++ toolbox ceph osd stat 2022-03-20T15:01:08.627 INFO:teuthology.orchestra.run.smithi007.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-20T15:01:08.627 INFO:teuthology.orchestra.run.smithi007.stderr:++ cut -f3 -d ' ' 2022-03-20T15:01:08.640 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:08.638491 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:01:08.692 INFO:teuthology.orchestra.run.smithi007.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-20T15:01:08.733 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:08.731770 I | op-osd: updating OSD 4 on PVC "smithi087-lv-1-data-0p7bb2" 2022-03-20T15:01:08.734 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:08.731827 I | op-osd: OSD will have its main bluestore block on "smithi087-lv-1-data-0p7bb2" 2022-03-20T15:01:09.144 INFO:teuthology.orchestra.run.smithi007.stderr:+ num_osd=11 2022-03-20T15:01:09.144 INFO:teuthology.orchestra.run.smithi007.stderr:+ sleep 30s 2022-03-20T15:01:09.452 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:09.449831 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:01:09.533 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:09.530902 I | op-osd: updating OSD 5 on PVC "smithi007-lv-4-data-0snpfz" 2022-03-20T15:01:09.533 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:09.530935 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-4-data-0snpfz" 2022-03-20T15:01:10.240 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:10.238859 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:01:10.369 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:10.367234 I | op-osd: updating OSD 6 on PVC "smithi087-lv-4-data-0vssdn" 2022-03-20T15:01:10.369 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:10.367259 I | op-osd: OSD will have its main bluestore block on "smithi087-lv-4-data-0vssdn" 2022-03-20T15:01:11.051 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:11.048668 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:01:11.132 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:11.130302 I | op-osd: updating OSD 7 on PVC "smithi007-lv-1-data-0fltgg" 2022-03-20T15:01:11.132 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:11.130325 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-1-data-0fltgg" 2022-03-20T15:01:11.899 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:11.896944 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:01:11.937 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:11.931434 I | op-osd: updating OSD 8 on PVC "smithi007-lv-3-data-0wb6kn" 2022-03-20T15:01:11.937 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:11.931471 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-3-data-0wb6kn" 2022-03-20T15:01:12.640 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:12.638572 I | op-osd: skipping osd checks for ok-to-stop 2022-03-20T15:01:12.744 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:12.742239 I | op-osd: updating OSD 9 on PVC "smithi007-lv-2-data-0z6lbn" 2022-03-20T15:01:12.744 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:12.742279 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-2-data-0z6lbn" 2022-03-20T15:01:14.559 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:14.557350 I | op-osd: OSD orchestration status for node smithi057-lv-2-data-04c4hs is "orchestrating" 2022-03-20T15:01:14.950 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:14.948821 I | op-osd: OSD orchestration status for PVC smithi057-lv-2-data-04c4hs is "orchestrating" 2022-03-20T15:01:21.075 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:21.072676 I | op-osd: OSD orchestration status for PVC smithi057-lv-2-data-04c4hs is "completed" 2022-03-20T15:01:21.076 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:21.072712 I | op-osd: creating OSD 0 on PVC "smithi057-lv-2-data-04c4hs" 2022-03-20T15:01:21.076 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:21.072729 I | op-osd: OSD will have its main bluestore block on "smithi057-lv-2-data-04c4hs" 2022-03-20T15:01:21.685 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:21.683740 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-20T15:01:21.686 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:01:21.683760 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-20T15:01:39.145 INFO:teuthology.orchestra.run.smithi007.stderr:+ '[' 11 '!=' 12 ']' 2022-03-20T15:01:39.146 INFO:teuthology.orchestra.run.smithi007.stdout:waiting for osd to come back up 2022-03-20T15:01:39.147 INFO:teuthology.orchestra.run.smithi007.stderr:+ echo 'waiting for osd to come back up' 2022-03-20T15:01:39.147 INFO:teuthology.orchestra.run.smithi007.stderr:++ toolbox ceph osd stat 2022-03-20T15:01:39.147 INFO:teuthology.orchestra.run.smithi007.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-20T15:01:39.148 INFO:teuthology.orchestra.run.smithi007.stderr:++ cut -f3 -d ' ' 2022-03-20T15:01:39.224 INFO:teuthology.orchestra.run.smithi007.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-20T15:01:39.813 INFO:teuthology.orchestra.run.smithi007.stderr:+ num_osd=11 2022-03-20T15:01:39.814 INFO:teuthology.orchestra.run.smithi007.stderr:+ sleep 30s 2022-03-20T15:02:09.815 INFO:teuthology.orchestra.run.smithi007.stderr:+ '[' 11 '!=' 12 ']' 2022-03-20T15:02:09.815 INFO:teuthology.orchestra.run.smithi007.stderr:+ echo 'waiting for osd to come back up' 2022-03-20T15:02:09.816 INFO:teuthology.orchestra.run.smithi007.stdout:waiting for osd to come back up 2022-03-20T15:02:09.816 INFO:teuthology.orchestra.run.smithi007.stderr:++ toolbox ceph osd stat 2022-03-20T15:02:09.817 INFO:teuthology.orchestra.run.smithi007.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-20T15:02:09.817 INFO:teuthology.orchestra.run.smithi007.stderr:++ cut -f3 -d ' ' 2022-03-20T15:02:09.946 INFO:teuthology.orchestra.run.smithi007.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-20T15:02:10.426 INFO:teuthology.orchestra.run.smithi007.stderr:+ num_osd=12 2022-03-20T15:02:10.426 INFO:teuthology.orchestra.run.smithi007.stderr:+ sleep 30s 2022-03-20T15:02:40.428 INFO:teuthology.orchestra.run.smithi007.stderr:+ '[' 12 '!=' 12 ']' 2022-03-20T15:02:40.430 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-20T15:02:40.443 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch status 2022-03-20T15:02:41.262 INFO:teuthology.orchestra.run.smithi007.stdout:Backend: rook 2022-03-20T15:02:41.263 INFO:teuthology.orchestra.run.smithi007.stdout:Available: Yes 2022-03-20T15:02:41.288 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ps 2022-03-20T15:02:41.871 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-20T15:02:41.872 INFO:teuthology.orchestra.run.smithi007.stdout:crashcollector.crash smithi007 running (4m) 0s ago 4m - - 7c0c85ada2e4 2022-03-20T15:02:41.872 INFO:teuthology.orchestra.run.smithi007.stdout:crashcollector.crash smithi057 running (5m) 0s ago 5m - - 7c0c85ada2e4 2022-03-20T15:02:41.872 INFO:teuthology.orchestra.run.smithi007.stdout:crashcollector.crash smithi087 running (5m) 0s ago 5m - - 7c0c85ada2e4 2022-03-20T15:02:41.873 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.a smithi007 running (5m) 0s ago 5m - - 7c0c85ada2e4 2022-03-20T15:02:41.873 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi087 running (8m) 0s ago 8m - - 7c0c85ada2e4 2022-03-20T15:02:41.873 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi057 running (7m) 0s ago 7m - - 7c0c85ada2e4 2022-03-20T15:02:41.873 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (7m) 0s ago 7m - - 7c0c85ada2e4 2022-03-20T15:02:41.873 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi057 running (80s) 0s ago 80s - - 7c0c85ada2e4 2022-03-20T15:02:41.874 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi057 running (4m) 0s ago 4m - - 7c0c85ada2e4 2022-03-20T15:02:41.874 INFO:teuthology.orchestra.run.smithi007.stdout:osd.10 smithi087 running (3m) 0s ago 3m - - 7c0c85ada2e4 2022-03-20T15:02:41.874 INFO:teuthology.orchestra.run.smithi007.stdout:osd.11 smithi087 running (3m) 0s ago 3m - - 7c0c85ada2e4 2022-03-20T15:02:41.874 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi057 running (4m) 0s ago 4m - - 7c0c85ada2e4 2022-03-20T15:02:41.874 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi057 running (4m) 0s ago 4m - - 7c0c85ada2e4 2022-03-20T15:02:41.875 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi087 running (4m) 0s ago 4m - - 7c0c85ada2e4 2022-03-20T15:02:41.875 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi007 running (4m) 0s ago 4m - - 7c0c85ada2e4 2022-03-20T15:02:41.875 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi087 running (4m) 0s ago 4m - - 7c0c85ada2e4 2022-03-20T15:02:41.875 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi007 running (4m) 0s ago 4m - - 7c0c85ada2e4 2022-03-20T15:02:41.876 INFO:teuthology.orchestra.run.smithi007.stdout:osd.8 smithi007 running (4m) 0s ago 4m - - 7c0c85ada2e4 2022-03-20T15:02:41.876 INFO:teuthology.orchestra.run.smithi007.stdout:osd.9 smithi007 running (4m) 0s ago 4m - - 7c0c85ada2e4 2022-03-20T15:02:41.896 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls 2022-03-20T15:02:42.613 INFO:teuthology.orchestra.run.smithi007.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-20T15:02:42.613 INFO:teuthology.orchestra.run.smithi007.stdout:crash 3/3 0s ago 5m * 2022-03-20T15:02:42.614 INFO:teuthology.orchestra.run.smithi007.stdout:mgr 1/1 0s ago 5m count:1 2022-03-20T15:02:42.614 INFO:teuthology.orchestra.run.smithi007.stdout:mon 3/3 0s ago 8m count:3 2022-03-20T15:02:42.614 INFO:teuthology.orchestra.run.smithi007.stdout:osd 24 0s ago 4m 2022-03-20T15:02:42.614 INFO:teuthology.orchestra.run.smithi007.stdout:osd.all-available-devices 0 0s ago - * 2022-03-20T15:02:42.640 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch host ls 2022-03-20T15:02:43.320 INFO:teuthology.orchestra.run.smithi007.stdout:HOST ADDR LABELS STATUS 2022-03-20T15:02:43.320 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 172.21.15.7/smithi007 2022-03-20T15:02:43.320 INFO:teuthology.orchestra.run.smithi007.stdout:smithi057 172.21.15.57/smithi057 2022-03-20T15:02:43.321 INFO:teuthology.orchestra.run.smithi007.stdout:smithi087 172.21.15.87/smithi087 2022-03-20T15:02:43.321 INFO:teuthology.orchestra.run.smithi007.stdout:3 hosts in cluster 2022-03-20T15:02:43.341 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch device ls 2022-03-20T15:02:43.972 INFO:teuthology.orchestra.run.smithi007.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-20T15:02:43.972 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-20T15:02:43.972 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-20T15:02:43.973 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-20T15:02:43.973 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-20T15:02:43.973 INFO:teuthology.orchestra.run.smithi007.stdout:smithi057 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-20T15:02:43.973 INFO:teuthology.orchestra.run.smithi007.stdout:smithi057 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-20T15:02:43.974 INFO:teuthology.orchestra.run.smithi007.stdout:smithi057 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-20T15:02:43.974 INFO:teuthology.orchestra.run.smithi007.stdout:smithi057 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-20T15:02:43.974 INFO:teuthology.orchestra.run.smithi007.stdout:smithi087 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-20T15:02:43.974 INFO:teuthology.orchestra.run.smithi007.stdout:smithi087 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-20T15:02:43.974 INFO:teuthology.orchestra.run.smithi007.stdout:smithi087 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-20T15:02:43.975 INFO:teuthology.orchestra.run.smithi007.stdout:smithi087 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-20T15:02:43.993 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch apply rgw foo 2022-03-20T15:02:44.580 INFO:teuthology.orchestra.run.smithi007.stdout:Created 2022-03-20T15:02:44.582 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:02:44.580666 I | ceph-spec: adding finalizer "cephobjectstore.ceph.rook.io" on "foo" 2022-03-20T15:02:44.606 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch apply mds foo 2022-03-20T15:02:44.663 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:02:44.660959 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:02:44.663 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:02:44.661034 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e... 2022-03-20T15:02:45.209 INFO:teuthology.orchestra.run.smithi007.stdout:Created 2022-03-20T15:02:45.211 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:02:45.209275 I | ceph-spec: adding finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-20T15:02:45.229 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch apply rbd-mirror 2022-03-20T15:02:45.233 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:02:45.232133 W | ceph-file-controller: failed to set filesystem "foo" status to "". failed to update object "rook-ceph/foo" status: Operation cannot be fulfilled on cephfilesystems.ceph.rook.io "foo": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:02:45.245 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:02:45.243723 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:02:45.245 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:02:45.243790 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e... 2022-03-20T15:02:45.838 INFO:teuthology.orchestra.run.smithi007.stdout:Success 2022-03-20T15:02:45.860 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch apply nfs foo --port 12777 2022-03-20T15:02:45.873 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:02:45.871822 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:02:45.874 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:02:45.871887 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e... 2022-03-20T15:02:50.973 INFO:teuthology.orchestra.run.smithi007.stdout:Unable to create NFS daemon, check logs for more traceback 2022-03-20T15:02:50.973 INFO:teuthology.orchestra.run.smithi007.stdout:'RookOrchestrator' object has no attribute 'open_ioctx' 2022-03-20T15:02:50.998 DEBUG:teuthology.run_tasks:Unwinding manager rook 2022-03-20T15:02:51.008 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:02:51.507 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:02:51.505350 I | ceph-spec: detected ceph image version: "17.0.0-11187 quincy" 2022-03-20T15:02:51.824 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:02:51.824 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:02:51.670487Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:02:51.670487Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:02:51.670487Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:02:51.670487Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:02:51.670487Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:02:51.670487Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_name": "rbd-mirror", "service_type": "rbd-mirror", "status": {"last_refresh": "2022-03-20T15:02:51.670487Z", "running": 0, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:02:51.670487Z", "running": 0, "size": 1}}] 2022-03-20T15:02:51.853 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch rm mds.foo 2022-03-20T15:02:51.925 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:02:51.923335 W | ceph-rbd-mirror-controller: (DEPRECATED) use of peer secret names in CephRBDMirror is deprecated. Please use CephBlockPool CR to configure peer secret names and import peers. 2022-03-20T15:02:51.926 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:02:51.923369 I | ceph-rbd-mirror-controller: configure rbd-mirroring with 1 workers 2022-03-20T15:02:51.926 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:02:51.923386 I | cephclient: getting or creating ceph auth key "client.rbd-mirror.a" 2022-03-20T15:02:52.423 INFO:teuthology.orchestra.run.smithi007.stdout:Removed cephfilesystems/foo 2022-03-20T15:02:52.441 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch rm rbd-mirror 2022-03-20T15:02:53.040 INFO:teuthology.orchestra.run.smithi007.stdout:Removed cephrbdmirrors/default-rbd-mirror 2022-03-20T15:02:53.058 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch rm rgw.foo 2022-03-20T15:02:53.372 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:02:53.370581 I | ceph-rbd-mirror-controller: "rook-ceph-rbd-mirror-a" deployment started 2022-03-20T15:02:53.653 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:02:53.650991 I | ceph-spec: object "rook-ceph-rbd-mirror-a" matched on delete, reconciling 2022-03-20T15:02:53.799 INFO:teuthology.orchestra.run.smithi007.stdout:Removed cephobjectstores/foo 2022-03-20T15:02:53.825 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:02:53.852 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:02:53.850701 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi087": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:02:53.873 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:02:53.871388 I | ceph-spec: detected ceph image version: "17.0.0-11187 quincy" 2022-03-20T15:02:53.956 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:02:53.954608 I | ceph-spec: detected ceph image version: "17.0.0-11187 quincy" 2022-03-20T15:02:54.807 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:02:54.808 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:14.000000Z", "last_refresh": "2022-03-20T15:02:54.606309Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:02:54.606309Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:02:54.606309Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:02:54.606309Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:02:54.606309Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:02:54.606309Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:02:54.606309Z", "running": 0, "size": 1}}] 2022-03-20T15:02:56.201 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:02:56.198710 I | ceph-file-controller: start running mdses for filesystem "foo" 2022-03-20T15:02:56.204 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:02:56.202428 I | ceph-object-controller: reconciling object store deployments 2022-03-20T15:02:56.249 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:02:56.247138 I | ceph-object-controller: ceph object store gateway service running at 10.105.153.12 2022-03-20T15:02:56.249 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:02:56.247167 I | ceph-object-controller: reconciling object store pools 2022-03-20T15:02:56.602 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:02:56.600338 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-20T15:02:56.679 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:02:56.677166 I | cephclient: getting or creating ceph auth key "mds.foo-a" 2022-03-20T15:02:57.585 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:02:57.583428 I | op-mds: setting mds config flags 2022-03-20T15:02:57.586 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:02:57.583474 I | op-config: setting "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-20T15:02:58.081 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:02:58.078871 I | op-config: successfully set "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-20T15:02:58.089 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:02:58.087299 I | cephclient: getting or creating ceph auth key "mds.foo-b" 2022-03-20T15:02:58.839 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:02:58.828194 I | op-mds: setting mds config flags 2022-03-20T15:02:58.839 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:02:58.828318 I | op-config: setting "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-20T15:03:02.513 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:02.510834 I | op-config: successfully set "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-20T15:03:04.839 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:03:05.669 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:05.666837 I | ceph-file-controller: creating filesystem "foo" 2022-03-20T15:03:05.674 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:05.668343 I | cephclient: reconciling replicated pool foo.rgw.buckets.index succeeded 2022-03-20T15:03:07.443 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:07.441523 I | cephclient: reconciling replicated pool foo.rgw.meta succeeded 2022-03-20T15:03:07.447 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:07.443178 I | cephclient: reconciling replicated pool foo.rgw.buckets.non-ec succeeded 2022-03-20T15:03:07.447 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:07.443693 I | cephclient: reconciling replicated pool foo.rgw.log succeeded 2022-03-20T15:03:07.455 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:07.451628 I | cephclient: reconciling replicated pool foo.rgw.control succeeded 2022-03-20T15:03:09.037 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:03:09.037 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:57:16.000000Z", "last_refresh": "2022-03-20T15:03:08.870707Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:03:08.870707Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:03:08.870707Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:03:08.870707Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:03:08.870707Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:03:08.870707Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:03:08.870707Z", "running": 0, "size": 1}}] 2022-03-20T15:03:10.224 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:10.222006 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool ".rgw.root": failed to tag pool ".rgw.root" for application "rook-ceph-rgw": failed to get application for pool ".rgw.root": failed to get current application for pool .rgw.root: exit status 2 2022-03-20T15:03:10.246 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:10.243899 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:03:11.887 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:11.884826 I | cephclient: reconciling replicated pool foo-metadata succeeded 2022-03-20T15:03:13.203 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:13.200203 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:03:13.189+0000 7fa3a37c0380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:03:13.203 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:03:13.189+0000 7fa3a37c0380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:03:13.203 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:03:13.189+0000 7fa3a37c0380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:03:13.203 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:03:13.222 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:13.220414 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:03:13.814 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:13.811589 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:03:13.801+0000 7ff42005f380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:03:13.814 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:03:13.801+0000 7ff42005f380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:03:13.814 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:03:13.801+0000 7ff42005f380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:03:13.815 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:03:13.848 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:13.845730 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:03:14.438 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:14.435702 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:03:14.425+0000 7f3d1e7af380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:03:14.439 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:03:14.425+0000 7f3d1e7af380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:03:14.439 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:03:14.425+0000 7f3d1e7af380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:03:14.439 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:03:14.493 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:14.491623 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:03:15.095 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:15.092505 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:03:15.081+0000 7f8295609380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:03:15.095 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:03:15.081+0000 7f8295609380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:03:15.095 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:03:15.081+0000 7f8295609380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:03:15.096 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:03:15.129 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:15.127490 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:03:15.764 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:15.761346 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:03:15.753+0000 7f225125c380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:03:15.764 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:03:15.753+0000 7f225125c380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:03:15.764 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:03:15.753+0000 7f225125c380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:03:15.764 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:03:15.830 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:15.827837 I | cephclient: reconciling replicated pool foo-data0 succeeded 2022-03-20T15:03:15.833 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:15.828731 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:03:16.465 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:16.462550 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:03:16.453+0000 7f8fd44f7380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:03:16.465 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:03:16.453+0000 7f8fd44f7380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:03:16.465 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:03:16.453+0000 7f8fd44f7380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:03:16.466 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:03:16.473 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:16.471788 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:03:16.713 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:16.711610 I | cephclient: creating filesystem "foo" with metadata pool "foo-metadata" and data pools [foo-data0] 2022-03-20T15:03:17.125 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:17.122784 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:03:17.113+0000 7ff9eb3e6380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:03:17.126 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:03:17.113+0000 7ff9eb3e6380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:03:17.126 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:03:17.113+0000 7ff9eb3e6380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:03:17.126 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:03:17.138 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:17.136598 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:03:17.799 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:17.796514 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:03:17.785+0000 7fb6c2c4b380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:03:17.800 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:03:17.785+0000 7fb6c2c4b380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:03:17.800 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:03:17.785+0000 7fb6c2c4b380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:03:17.800 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:03:17.829 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:17.827796 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:03:18.435 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:18.432392 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:03:18.421+0000 7f3e555b3380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:03:18.435 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:03:18.421+0000 7f3e555b3380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:03:18.435 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:03:18.421+0000 7f3e555b3380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:03:18.436 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:03:18.603 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:18.601463 I | ceph-file-controller: created filesystem "foo" on 1 data pool(s) and metadata pool "foo-metadata" 2022-03-20T15:03:18.604 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:18.601494 I | cephclient: setting allow_standby_replay for filesystem "foo" 2022-03-20T15:03:19.057 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:03:19.133 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:19.131459 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:03:19.733 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:19.721115 W | ceph-file-controller: failed to set filesystem "foo" status to "". failed to update object "rook-ceph/foo" status: Operation cannot be fulfilled on cephfilesystems.ceph.rook.io "foo": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:03:19.756 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:19.754901 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:03:19.759 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:19.756747 I | ceph-file-controller: CephFilesystem "rook-ceph/foo" can be deleted safely. deleting CephFilesystem "rook-ceph/foo" 2022-03-20T15:03:19.859 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:19.856561 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:03:19.845+0000 7fa22ad63380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:03:19.859 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:03:19.845+0000 7fa22ad63380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:03:19.860 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:03:19.845+0000 7fa22ad63380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:03:19.860 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:03:20.064 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:03:20.064 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:03:19.669738Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:03:19.669738Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:03:19.669738Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:03:19.669738Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:03:19.669738Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:03:19.669738Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:03:19.669738Z", "running": 0, "size": 1}}] 2022-03-20T15:03:20.631 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:20.628927 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-20T15:03:21.255 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:21.252420 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-20T15:03:21.255 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:21.252452 I | op-mds: successfully deleted mds config for "mds.foo-a" in mon configuration database 2022-03-20T15:03:21.255 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:21.252460 I | cephclient: deleting ceph auth "mds.foo-a" 2022-03-20T15:03:21.821 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:21.818367 I | op-mds: successfully deleted mds CephX key for "mds.foo-a" 2022-03-20T15:03:22.255 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:22.253130 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-20T15:03:22.758 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:22.755635 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-20T15:03:22.758 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:22.755668 I | op-mds: successfully deleted mds config for "mds.foo-b" in mon configuration database 2022-03-20T15:03:22.759 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:22.755676 I | cephclient: deleting ceph auth "mds.foo-b" 2022-03-20T15:03:23.457 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:23.454744 I | op-mds: successfully deleted mds CephX key for "mds.foo-b" 2022-03-20T15:03:23.457 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:23.454780 I | ceph-file-controller: downing filesystem "foo" 2022-03-20T15:03:24.622 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:24.620497 I | ceph-file-controller: downed filesystem "foo" 2022-03-20T15:03:25.013 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:25.011568 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:03:26.032 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:26.029728 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:03:26.021+0000 7f4d717b9380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:03:26.032 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:03:26.021+0000 7f4d717b9380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:03:26.033 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:03:26.021+0000 7f4d717b9380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:03:26.033 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:03:28.829 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:28.825255 I | cephclient: no images/snapshosts present in pool "foo-metadata" 2022-03-20T15:03:28.831 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:28.825283 I | cephclient: purging pool "foo-metadata" (id=8) 2022-03-20T15:03:30.088 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:03:30.810 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:03:30.810 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:03:30.619336Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:03:30.619336Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:03:30.619336Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:03:30.619336Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:03:30.619336Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:03:30.619336Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:03:30.619336Z", "running": 0, "size": 1}}] 2022-03-20T15:03:31.327 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:31.325296 I | cephclient: purge completed for pool "foo-metadata" 2022-03-20T15:03:31.733 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:31.730710 I | cephclient: no images/snapshosts present in pool "foo-data0" 2022-03-20T15:03:31.733 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:31.730740 I | cephclient: purging pool "foo-data0" (id=10) 2022-03-20T15:03:34.038 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:34.034559 I | cephclient: purge completed for pool "foo-data0" 2022-03-20T15:03:34.038 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:34.034642 I | ceph-spec: removing finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-20T15:03:34.904 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:34.901768 I | ceph-spec: object "rook-ceph-mds-foo-b-keyring" matched on delete, reconciling 2022-03-20T15:03:34.905 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:34.901910 I | ceph-spec: object "rook-ceph-mds-foo-a-keyring" matched on delete, reconciling 2022-03-20T15:03:34.908 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:34.903697 I | ceph-spec: object "rook-ceph-mds-foo-b" matched on delete, reconciling 2022-03-20T15:03:34.909 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:34.903891 I | ceph-spec: object "rook-ceph-mds-foo-a" matched on delete, reconciling 2022-03-20T15:03:36.281 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:36.278942 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:03:38.118 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:38.116303 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:03:38.105+0000 7f1ed30e2380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:03:38.119 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:03:38.105+0000 7f1ed30e2380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:03:38.119 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:03:38.105+0000 7f1ed30e2380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:03:38.119 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:03:38.297 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:38.295676 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi057": the object has been modified; please apply your changes to the latest version and try again 2022-03-20T15:03:40.835 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:03:41.662 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:03:41.662 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:03:41.481130Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:03:41.481130Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:03:41.481130Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:03:41.481130Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:03:41.481130Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:03:41.481130Z", "running": 0, "size": 1}}] 2022-03-20T15:03:51.686 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:03:52.418 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:03:52.418 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:03:52.245072Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:03:52.245072Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:03:52.245072Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:03:52.245072Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:03:52.245072Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:03:52.245072Z", "running": 0, "size": 1}}] 2022-03-20T15:03:58.609 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:58.607067 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:03:59.347 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:03:59.344864 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:03:59.332+0000 7fac6f961380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:03:59.347 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:03:59.332+0000 7fac6f961380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:03:59.348 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:03:59.332+0000 7fac6f961380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:03:59.348 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:04:02.444 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:04:03.188 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:04:03.189 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:04:02.984879Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:04:02.984879Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:04:02.984879Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:04:02.984879Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:04:02.984879Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:04:02.984879Z", "running": 0, "size": 1}}] 2022-03-20T15:04:13.215 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:04:13.984 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:04:13.984 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:04:13.778859Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:04:13.778859Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:04:13.778859Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:04:13.778859Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:04:13.778859Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:04:13.778859Z", "running": 0, "size": 1}}] 2022-03-20T15:04:24.012 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:04:24.706 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:04:24.706 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:04:24.541584Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:04:24.541584Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:04:24.541584Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:04:24.541584Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:04:24.541584Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:04:24.541584Z", "running": 0, "size": 1}}] 2022-03-20T15:04:34.733 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:04:35.567 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:04:35.568 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:04:35.276101Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:04:35.276101Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:04:35.276101Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:04:35.276101Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:04:35.276101Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:04:35.276101Z", "running": 0, "size": 1}}] 2022-03-20T15:04:40.315 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:04:40.313015 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:04:41.305 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:04:41.302615 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:04:41.287+0000 7fbdc4296380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:04:41.305 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:04:41.287+0000 7fbdc4296380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:04:41.305 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:04:41.287+0000 7fbdc4296380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:04:41.306 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:04:45.617 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:04:46.451 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:04:46.451 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:04:46.184563Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:04:46.184563Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:04:46.184563Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:04:46.184563Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:04:46.184563Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:04:46.184563Z", "running": 0, "size": 1}}] 2022-03-20T15:04:56.482 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:04:57.221 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:04:57.222 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:04:57.043886Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:04:57.043886Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:04:57.043886Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:04:57.043886Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:04:57.043886Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:04:57.043886Z", "running": 0, "size": 1}}] 2022-03-20T15:05:07.245 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:05:08.342 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:05:08.343 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:05:07.942012Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:05:07.942012Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:05:07.942012Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:05:07.942012Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:05:07.942012Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:05:07.942012Z", "running": 0, "size": 1}}] 2022-03-20T15:05:18.364 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:05:19.097 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:05:19.098 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:05:18.921092Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:05:18.921092Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:05:18.921092Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:05:18.921092Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:05:18.921092Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:05:18.921092Z", "running": 0, "size": 1}}] 2022-03-20T15:05:29.124 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:05:29.892 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:05:29.892 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:05:29.723372Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:05:29.723372Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:05:29.723372Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:05:29.723372Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:05:29.723372Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:05:29.723372Z", "running": 0, "size": 1}}] 2022-03-20T15:05:39.919 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:05:40.686 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:05:40.686 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:05:40.486943Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:05:40.486943Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:05:40.486943Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:05:40.486943Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:05:40.486943Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:05:40.486943Z", "running": 0, "size": 1}}] 2022-03-20T15:05:50.713 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:05:51.419 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:05:51.420 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:05:51.248864Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:05:51.248864Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:05:51.248864Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:05:51.248864Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:05:51.248864Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:05:51.248864Z", "running": 0, "size": 1}}] 2022-03-20T15:06:01.446 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:06:02.235 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:06:02.235 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:06:02.041569Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:06:02.041569Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:06:02.041569Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:06:02.041569Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:06:02.041569Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:06:02.041569Z", "running": 0, "size": 1}}] 2022-03-20T15:06:03.236 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:06:03.234014 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:06:03.898 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:06:03.895256 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:06:03.886+0000 7f4be355a380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:06:03.898 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:06:03.886+0000 7f4be355a380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:06:03.898 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:06:03.886+0000 7f4be355a380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:06:03.899 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:06:12.264 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:06:13.233 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:06:13.234 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:06:12.799830Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:06:12.799830Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:06:12.799830Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:06:12.799830Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:06:12.799830Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:06:12.799830Z", "running": 0, "size": 1}}] 2022-03-20T15:06:23.268 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:06:23.990 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:06:23.990 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:06:23.819391Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:06:23.819391Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:06:23.819391Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:06:23.819391Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:06:23.819391Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:06:23.819391Z", "running": 0, "size": 1}}] 2022-03-20T15:06:34.020 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:06:34.775 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:06:34.776 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:06:34.604581Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:06:34.604581Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:06:34.604581Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:06:34.604581Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:06:34.604581Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:06:34.604581Z", "running": 0, "size": 1}}] 2022-03-20T15:06:44.802 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:06:45.537 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:06:45.537 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:06:45.357568Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:06:45.357568Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:06:45.357568Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:06:45.357568Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:06:45.357568Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:06:45.357568Z", "running": 0, "size": 1}}] 2022-03-20T15:06:55.559 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:06:56.261 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:06:56.262 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:06:56.078916Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:06:56.078916Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:06:56.078916Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:06:56.078916Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:06:56.078916Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:06:56.078916Z", "running": 0, "size": 1}}] 2022-03-20T15:07:06.291 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:07:07.007 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:07:07.007 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:07:06.835328Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:07:06.835328Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:07:06.835328Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:07:06.835328Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:07:06.835328Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:07:06.835328Z", "running": 0, "size": 1}}] 2022-03-20T15:07:17.032 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:07:17.771 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:07:17.771 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:07:17.595913Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:07:17.595913Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:07:17.595913Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:07:17.595913Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:07:17.595913Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:07:17.595913Z", "running": 0, "size": 1}}] 2022-03-20T15:07:27.798 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:07:28.565 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:07:28.565 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:07:28.370326Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:07:28.370326Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:07:28.370326Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:07:28.370326Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:07:28.370326Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:07:28.370326Z", "running": 0, "size": 1}}] 2022-03-20T15:07:38.594 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:07:39.337 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:07:39.337 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:07:39.130190Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:07:39.130190Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:07:39.130190Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:07:39.130190Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:07:39.130190Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:07:39.130190Z", "running": 0, "size": 1}}] 2022-03-20T15:07:49.364 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:07:50.103 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:07:50.103 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:07:49.875691Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:07:49.875691Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:07:49.875691Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:07:49.875691Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:07:49.875691Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:07:49.875691Z", "running": 0, "size": 1}}] 2022-03-20T15:08:00.127 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:08:00.818 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:08:00.819 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:08:00.637744Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:08:00.637744Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:08:00.637744Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:08:00.637744Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:08:00.637744Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:08:00.637744Z", "running": 0, "size": 1}}] 2022-03-20T15:08:10.852 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:08:11.619 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:08:11.619 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:08:11.414257Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:08:11.414257Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:08:11.414257Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:08:11.414257Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:08:11.414257Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:08:11.414257Z", "running": 0, "size": 1}}] 2022-03-20T15:08:21.650 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:08:22.353 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:08:22.354 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:08:22.166947Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:08:22.166947Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:08:22.166947Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:08:22.166947Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:08:22.166947Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:08:22.166947Z", "running": 0, "size": 1}}] 2022-03-20T15:08:32.382 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:08:33.119 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:08:33.120 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:08:32.917463Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:08:32.917463Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:08:32.917463Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:08:32.917463Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:08:32.917463Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:08:32.917463Z", "running": 0, "size": 1}}] 2022-03-20T15:08:43.148 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:08:43.888 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:08:43.888 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:08:43.709175Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:08:43.709175Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:08:43.709175Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:08:43.709175Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:08:43.709175Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:08:43.709175Z", "running": 0, "size": 1}}] 2022-03-20T15:08:47.749 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:08:47.747059 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:08:48.473 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:08:48.470011 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:08:48.459+0000 7f61a3e23380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:08:48.473 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:08:48.459+0000 7f61a3e23380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:08:48.473 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:08:48.459+0000 7f61a3e23380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:08:48.473 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:08:53.915 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:08:54.624 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:08:54.624 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:08:54.443331Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:08:54.443331Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:08:54.443331Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:08:54.443331Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:08:54.443331Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:08:54.443331Z", "running": 0, "size": 1}}] 2022-03-20T15:09:04.652 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:09:05.381 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:09:05.381 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:09:05.207675Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:09:05.207675Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:09:05.207675Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:09:05.207675Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:09:05.207675Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:09:05.207675Z", "running": 0, "size": 1}}] 2022-03-20T15:09:15.409 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:09:16.137 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:09:16.137 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:09:15.957862Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:09:15.957862Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:09:15.957862Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:09:15.957862Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:09:15.957862Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:09:15.957862Z", "running": 0, "size": 1}}] 2022-03-20T15:09:26.183 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:09:26.843 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:09:26.843 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:09:26.695315Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:09:26.695315Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:09:26.695315Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:09:26.695315Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:09:26.695315Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:09:26.695315Z", "running": 0, "size": 1}}] 2022-03-20T15:09:36.868 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:09:37.616 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:09:37.617 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:09:37.435195Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:09:37.435195Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:09:37.435195Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:09:37.435195Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:09:37.435195Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:09:37.435195Z", "running": 0, "size": 1}}] 2022-03-20T15:09:47.649 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:09:48.382 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:09:48.383 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:09:48.219317Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:09:48.219317Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:09:48.219317Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:09:48.219317Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:09:48.219317Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:09:48.219317Z", "running": 0, "size": 1}}] 2022-03-20T15:09:58.415 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:09:59.103 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:09:59.103 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:09:58.928057Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:09:58.928057Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:09:58.928057Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:09:58.928057Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:09:58.928057Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:09:58.928057Z", "running": 0, "size": 1}}] 2022-03-20T15:10:09.134 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:10:09.885 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:10:09.886 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:10:09.710604Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:10:09.710604Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:10:09.710604Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:10:09.710604Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:10:09.710604Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:10:09.710604Z", "running": 0, "size": 1}}] 2022-03-20T15:10:19.914 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:10:20.655 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:10:20.655 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:10:20.488174Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:10:20.488174Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:10:20.488174Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:10:20.488174Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:10:20.488174Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:10:20.488174Z", "running": 0, "size": 1}}] 2022-03-20T15:10:30.678 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:10:31.401 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:10:31.401 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:10:31.211218Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:10:31.211218Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:10:31.211218Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:10:31.211218Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:10:31.211218Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:10:31.211218Z", "running": 0, "size": 1}}] 2022-03-20T15:10:41.431 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:10:42.166 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:10:42.166 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:10:41.991109Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:10:41.991109Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:10:41.991109Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:10:41.991109Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:10:41.991109Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:10:41.991109Z", "running": 0, "size": 1}}] 2022-03-20T15:10:52.193 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:10:52.907 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:10:52.908 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:10:52.736010Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:10:52.736010Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:10:52.736010Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:10:52.736010Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:10:52.736010Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:10:52.736010Z", "running": 0, "size": 1}}] 2022-03-20T15:11:02.934 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:11:03.656 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:11:03.656 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:11:03.460266Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:11:03.460266Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:11:03.460266Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:11:03.460266Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:11:03.460266Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:11:03.460266Z", "running": 0, "size": 1}}] 2022-03-20T15:11:13.684 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:11:14.723 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:11:14.723 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:11:14.260016Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:11:14.260016Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:11:14.260016Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:11:14.260016Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:11:14.260016Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:11:14.260016Z", "running": 0, "size": 1}}] 2022-03-20T15:11:24.749 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:11:25.473 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:11:25.474 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:11:25.266990Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:11:25.266990Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:11:25.266990Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:11:25.266990Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:11:25.266990Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:11:25.266990Z", "running": 0, "size": 1}}] 2022-03-20T15:11:35.500 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:11:36.227 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:11:36.227 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:11:36.045455Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:11:36.045455Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:11:36.045455Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:11:36.045455Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:11:36.045455Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:11:36.045455Z", "running": 0, "size": 1}}] 2022-03-20T15:11:46.256 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:11:47.040 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:11:47.041 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:11:46.768299Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:11:46.768299Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:11:46.768299Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:11:46.768299Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:11:46.768299Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:11:46.768299Z", "running": 0, "size": 1}}] 2022-03-20T15:11:57.070 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:11:57.775 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:11:57.776 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:11:57.579730Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:11:57.579730Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:11:57.579730Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:11:57.579730Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:11:57.579730Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:11:57.579730Z", "running": 0, "size": 1}}] 2022-03-20T15:12:07.805 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:12:08.511 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:12:08.512 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:12:08.349700Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:12:08.349700Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:12:08.349700Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:12:08.349700Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:12:08.349700Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:12:08.349700Z", "running": 0, "size": 1}}] 2022-03-20T15:12:18.537 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:12:19.243 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:12:19.243 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:12:19.078730Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:12:19.078730Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:12:19.078730Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:12:19.078730Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:12:19.078730Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:12:19.078730Z", "running": 0, "size": 1}}] 2022-03-20T15:12:29.269 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:12:30.037 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:12:30.038 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:12:29.866886Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:12:29.866886Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:12:29.866886Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:12:29.866886Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:12:29.866886Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:12:29.866886Z", "running": 0, "size": 1}}] 2022-03-20T15:12:40.064 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:12:40.891 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:12:40.891 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:12:40.650819Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:12:40.650819Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:12:40.650819Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:12:40.650819Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:12:40.650819Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:12:40.650819Z", "running": 0, "size": 1}}] 2022-03-20T15:12:50.920 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:12:51.653 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:12:51.653 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:12:51.460933Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:12:51.460933Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:12:51.460933Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:12:51.460933Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:12:51.460933Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:12:51.460933Z", "running": 0, "size": 1}}] 2022-03-20T15:13:01.679 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:13:02.419 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:13:02.420 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:13:02.182899Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:13:02.182899Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:13:02.182899Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:13:02.182899Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:13:02.182899Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:13:02.182899Z", "running": 0, "size": 1}}] 2022-03-20T15:13:12.445 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:13:15.559 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:13:15.559 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:13:15.386735Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:13:15.386735Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:13:15.386735Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:13:15.386735Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:13:15.386735Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:13:15.386735Z", "running": 0, "size": 1}}] 2022-03-20T15:13:25.593 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:13:26.325 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:13:26.326 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:13:26.148489Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:13:26.148489Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:13:26.148489Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:13:26.148489Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:13:26.148489Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:13:26.148489Z", "running": 0, "size": 1}}] 2022-03-20T15:13:36.353 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:13:37.044 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:13:37.045 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:13:36.870004Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:13:36.870004Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:13:36.870004Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:13:36.870004Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:13:36.870004Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:13:36.870004Z", "running": 0, "size": 1}}] 2022-03-20T15:13:47.070 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:13:47.849 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:13:47.850 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:13:47.641040Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:13:47.641040Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:13:47.641040Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:13:47.641040Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:13:47.641040Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:13:47.641040Z", "running": 0, "size": 1}}] 2022-03-20T15:13:57.875 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:13:58.598 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:13:58.599 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:13:58.418763Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:13:58.418763Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:13:58.418763Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:13:58.418763Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:13:58.418763Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:13:58.418763Z", "running": 0, "size": 1}}] 2022-03-20T15:14:08.627 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:14:09.430 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:14:09.431 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:14:09.221375Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:14:09.221375Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:14:09.221375Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:14:09.221375Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:14:09.221375Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:14:09.221375Z", "running": 0, "size": 1}}] 2022-03-20T15:14:16.158 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:14:16.156165 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:14:16.942 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:14:16.940199 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:14:16.930+0000 7f6ba76fb380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:14:16.943 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:14:16.930+0000 7f6ba76fb380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:14:16.943 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:14:16.930+0000 7f6ba76fb380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:14:16.943 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:14:19.455 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:14:20.223 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:14:20.223 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:14:19.984114Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:14:19.984114Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:14:19.984114Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:14:19.984114Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:14:19.984114Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:14:19.984114Z", "running": 0, "size": 1}}] 2022-03-20T15:14:30.251 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:14:30.960 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:14:30.961 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:14:30.786797Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:14:30.786797Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:14:30.786797Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:14:30.786797Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:14:30.786797Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:14:30.786797Z", "running": 0, "size": 1}}] 2022-03-20T15:14:40.993 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:14:41.760 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:14:41.760 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:14:41.597033Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:14:41.597033Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:14:41.597033Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:14:41.597033Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:14:41.597033Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:14:41.597033Z", "running": 0, "size": 1}}] 2022-03-20T15:14:51.787 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:14:52.521 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:14:52.521 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:14:52.308280Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:14:52.308280Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:14:52.308280Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:14:52.308280Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:14:52.308280Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:14:52.308280Z", "running": 0, "size": 1}}] 2022-03-20T15:15:02.547 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:15:03.257 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:15:03.258 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:15:03.080225Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:15:03.080225Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:15:03.080225Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:15:03.080225Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:15:03.080225Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:15:03.080225Z", "running": 0, "size": 1}}] 2022-03-20T15:15:13.284 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:15:14.047 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:15:14.047 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:15:13.854221Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:15:13.854221Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:15:13.854221Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:15:13.854221Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:15:13.854221Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:15:13.854221Z", "running": 0, "size": 1}}] 2022-03-20T15:15:24.075 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:15:24.777 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:15:24.777 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:15:24.598810Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:15:24.598810Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:15:24.598810Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:15:24.598810Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:15:24.598810Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:15:24.598810Z", "running": 0, "size": 1}}] 2022-03-20T15:15:34.803 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:15:35.531 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:15:35.531 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:15:35.354518Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:15:35.354518Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:15:35.354518Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:15:35.354518Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:15:35.354518Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:15:35.354518Z", "running": 0, "size": 1}}] 2022-03-20T15:15:45.558 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:15:46.295 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:15:46.295 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:15:46.108289Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:15:46.108289Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:15:46.108289Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:15:46.108289Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:15:46.108289Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:15:46.108289Z", "running": 0, "size": 1}}] 2022-03-20T15:15:56.348 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:15:57.057 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:15:57.058 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:15:56.875536Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:15:56.875536Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:15:56.875536Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:15:56.875536Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:15:56.875536Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:15:56.875536Z", "running": 0, "size": 1}}] 2022-03-20T15:16:07.082 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:16:07.821 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:16:07.822 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:16:07.657620Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:16:07.657620Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:16:07.657620Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:16:07.657620Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:16:07.657620Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:16:07.657620Z", "running": 0, "size": 1}}] 2022-03-20T15:16:17.849 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:16:18.579 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:16:18.579 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:16:18.403401Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:16:18.403401Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:16:18.403401Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:16:18.403401Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:16:18.403401Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:16:18.403401Z", "running": 0, "size": 1}}] 2022-03-20T15:16:28.607 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:16:29.347 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:16:29.348 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:16:29.176014Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:16:29.176014Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:16:29.176014Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:16:29.176014Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:16:29.176014Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:16:29.176014Z", "running": 0, "size": 1}}] 2022-03-20T15:16:39.375 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:16:40.123 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:16:40.123 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:16:39.928769Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:16:39.928769Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:16:39.928769Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:16:39.928769Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:16:39.928769Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:16:39.928769Z", "running": 0, "size": 1}}] 2022-03-20T15:16:50.151 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:16:50.878 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:16:50.878 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:16:50.696796Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:16:50.696796Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:16:50.696796Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:16:50.696796Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:16:50.696796Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:16:50.696796Z", "running": 0, "size": 1}}] 2022-03-20T15:17:00.904 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:17:01.982 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:17:01.982 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:17:01.453375Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:17:01.453375Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:17:01.453375Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:17:01.453375Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:17:01.453375Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:17:01.453375Z", "running": 0, "size": 1}}] 2022-03-20T15:17:12.009 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:17:12.737 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:17:12.738 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:17:12.534421Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:17:12.534421Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:17:12.534421Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:17:12.534421Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:17:12.534421Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:17:12.534421Z", "running": 0, "size": 1}}] 2022-03-20T15:17:22.764 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:17:23.489 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:17:23.489 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:17:23.325597Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:17:23.325597Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:17:23.325597Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:17:23.325597Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:17:23.325597Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:17:23.325597Z", "running": 0, "size": 1}}] 2022-03-20T15:17:33.513 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:17:34.263 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:17:34.264 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:17:34.082016Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:17:34.082016Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:17:34.082016Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:17:34.082016Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:17:34.082016Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:17:34.082016Z", "running": 0, "size": 1}}] 2022-03-20T15:17:44.291 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:17:45.029 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:17:45.030 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:17:44.826644Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:17:44.826644Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:17:44.826644Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:17:44.826644Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:17:44.826644Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:17:44.826644Z", "running": 0, "size": 1}}] 2022-03-20T15:17:55.064 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:17:55.782 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:17:55.783 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:17:55.613203Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:17:55.613203Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:17:55.613203Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:17:55.613203Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:17:55.613203Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:17:55.613203Z", "running": 0, "size": 1}}] 2022-03-20T15:18:05.808 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:18:06.577 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:18:06.577 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:18:06.409574Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:18:06.409574Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:18:06.409574Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:18:06.409574Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:18:06.409574Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:18:06.409574Z", "running": 0, "size": 1}}] 2022-03-20T15:18:16.604 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:18:17.305 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:18:17.305 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:18:17.135616Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:18:17.135616Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:18:17.135616Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:18:17.135616Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:18:17.135616Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:18:17.135616Z", "running": 0, "size": 1}}] 2022-03-20T15:18:27.331 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:18:28.118 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:18:28.118 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:18:27.919228Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:18:27.919228Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:18:27.919228Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:18:27.919228Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:18:27.919228Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:18:27.919228Z", "running": 0, "size": 1}}] 2022-03-20T15:18:38.143 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:18:38.823 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:18:38.824 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:18:38.674394Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:18:38.674394Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:18:38.674394Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:18:38.674394Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:18:38.674394Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:18:38.674394Z", "running": 0, "size": 1}}] 2022-03-20T15:18:48.852 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:18:49.556 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:18:49.557 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:18:49.385658Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:18:49.385658Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:18:49.385658Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:18:49.385658Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:18:49.385658Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:18:49.385658Z", "running": 0, "size": 1}}] 2022-03-20T15:18:59.615 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-64sd5 -- ceph orch ls -f json 2022-03-20T15:19:00.337 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-20T15:19:00.338 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:28.000000Z", "last_refresh": "2022-03-20T15:19:00.167377Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:56:47.000000Z", "last_refresh": "2022-03-20T15:19:00.167377Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:54:14.000000Z", "last_refresh": "2022-03-20T15:19:00.167377Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "7c0c85ada2e42f0b213c377d2268e8c6ecb4e75a5bdb3ce8cd150acb69ae64a3", "container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "created": "2022-03-20T14:58:25.000000Z", "last_refresh": "2022-03-20T15:19:00.167377Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-20T15:19:00.167377Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:e92561bf8a1bc96ca490936c6f46ac4673e1b26e", "last_refresh": "2022-03-20T15:19:00.167377Z", "running": 0, "size": 1}}] 2022-03-20T15:19:00.364 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-20T15:19:00.364 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks/rook.py", line 530, in ceph_config_keyring yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-20T15:19:00.365 INFO:tasks.rook:Cleaning up config and client.admin keyring 2022-03-20T15:19:00.365 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-20T15:19:00.380 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-20T15:19:00.396 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-20T15:19:00.412 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks/rook.py", line 478, in rook_post_config yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-20T15:19:00.413 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks/rook.py", line 442, in rook_toolbox yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-20T15:19:00.426 DEBUG:teuthology.orchestra.remote:smithi007:rook/deploy/examples/operator.yaml is 20KB 2022-03-20T15:19:00.442 DEBUG:teuthology.orchestra.run.smithi007:> kubectl delete -f rook/deploy/examples/toolbox.yaml 2022-03-20T15:19:00.505 INFO:teuthology.orchestra.run.smithi007.stdout:deployment.apps "rook-ceph-tools" deleted 2022-03-20T15:19:00.523 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks/rook.py", line 379, in rook_cluster yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e92561bf8a1bc96ca490936c6f46ac4673e1b26e/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-20T15:19:00.524 DEBUG:teuthology.orchestra.run.smithi007:> kubectl delete -f cluster.yaml 2022-03-20T15:19:00.687 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:00.684682 I | ceph-cluster-controller: CR "rook-ceph" is going be deleted, cancelling any ongoing orchestration 2022-03-20T15:19:00.687 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:00.684816 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-20T15:19:00.702 INFO:tasks.rook.operator.smithi007.stdout:I0320 15:19:00.684991 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-20T15:19:00.702 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:00.685230 I | operator: successfully started the controller-runtime manager 2022-03-20T15:19:00.702 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:00.694604 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-20T15:19:00.703 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:00.694628 I | operator: watching all namespaces for Ceph CRs 2022-03-20T15:19:00.703 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:00.694665 I | operator: setting up schemes 2022-03-20T15:19:00.703 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:00.697755 I | operator: setting up the controller-runtime manager 2022-03-20T15:19:00.704 INFO:teuthology.orchestra.run.smithi007.stdout:cephcluster.ceph.rook.io "rook-ceph" deleted 2022-03-20T15:19:00.905 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:00.901786 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-20T15:19:00.910 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:00.905567 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-20T15:19:00.910 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:00.905612 I | ceph-cluster-controller: successfully started 2022-03-20T15:19:00.911 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:00.905673 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-20T15:19:00.911 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:00.905704 I | ceph-crashcollector-controller: successfully started 2022-03-20T15:19:00.911 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:00.905747 I | ceph-block-pool-controller: successfully started 2022-03-20T15:19:00.911 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:00.905794 I | ceph-object-store-user-controller: successfully started 2022-03-20T15:19:00.911 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:00.905825 I | ceph-object-realm-controller: successfully started 2022-03-20T15:19:00.912 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:00.905845 I | ceph-object-zonegroup-controller: successfully started 2022-03-20T15:19:00.912 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:00.905859 I | ceph-object-zone-controller: successfully started 2022-03-20T15:19:00.912 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:00.906035 I | ceph-object-controller: successfully started 2022-03-20T15:19:00.914 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:00.906082 I | ceph-file-controller: successfully started 2022-03-20T15:19:00.914 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:00.906122 I | ceph-nfs-controller: successfully started 2022-03-20T15:19:00.914 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:00.906174 I | ceph-rbd-mirror-controller: successfully started 2022-03-20T15:19:00.915 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:00.906215 I | ceph-client-controller: successfully started 2022-03-20T15:19:00.915 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:00.906254 I | ceph-filesystem-mirror-controller: successfully started 2022-03-20T15:19:00.915 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:00.906292 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-20T15:19:00.915 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:00.906323 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-20T15:19:00.916 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:00.906358 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-20T15:19:00.916 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:00.906382 I | ceph-bucket-topic: successfully started 2022-03-20T15:19:00.916 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:00.906404 I | ceph-bucket-notification: successfully started 2022-03-20T15:19:00.916 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:00.906430 I | ceph-bucket-notification: successfully started 2022-03-20T15:19:00.916 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:00.906449 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-20T15:19:00.917 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:00.907579 I | operator: starting the controller-runtime manager 2022-03-20T15:19:01.012 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:01.010104 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-20T15:19:01.012 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:01.010169 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-20T15:19:01.012 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:01.010187 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-20T15:19:01.036 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:01.034672 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-20T15:19:01.075 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:01.073458 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:19:01.103 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:01.101346 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:19:01.114 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:01.113119 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:19:01.745 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:01.742948 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:19:01.733+0000 7f94c3aed380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:19:01.745 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:19:01.733+0000 7f94c3aed380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:19:01.746 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:19:01.733+0000 7f94c3aed380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:19:01.746 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:19:01.790 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:01.788185 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:19:02.443 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:02.440784 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:19:02.429+0000 7ff8ec573380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:19:02.443 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:19:02.429+0000 7ff8ec573380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:19:02.443 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:19:02.429+0000 7ff8ec573380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:19:02.444 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:19:02.501 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:02.499390 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:19:03.158 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:03.156422 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:19:03.145+0000 7f1280f86380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:19:03.159 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:19:03.145+0000 7f1280f86380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:19:03.159 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:19:03.145+0000 7f1280f86380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:19:03.159 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:19:03.186 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:03.184996 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:19:03.888 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:03.886009 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:19:03.873+0000 7fe3de48b380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:19:03.889 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:19:03.873+0000 7fe3de48b380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:19:03.889 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:19:03.873+0000 7fe3de48b380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:19:03.889 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:19:03.935 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:03.932542 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:19:04.655 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:04.653068 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:19:04.641+0000 7f062b869380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:19:04.655 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:19:04.641+0000 7f062b869380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:19:04.656 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:19:04.641+0000 7f062b869380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:19:04.656 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:19:04.742 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:04.740350 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:19:05.417 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:05.414683 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:19:05.401+0000 7f2f6eadb380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:19:05.417 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:19:05.401+0000 7f2f6eadb380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:19:05.417 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:19:05.401+0000 7f2f6eadb380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:19:05.418 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:19:05.586 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:05.584169 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:19:06.267 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:06.264619 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:19:06.253+0000 7ff083a60380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:19:06.267 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:19:06.253+0000 7ff083a60380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:19:06.267 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:19:06.253+0000 7ff083a60380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:19:06.268 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:19:06.596 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:06.594618 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:19:07.241 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:07.238895 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:19:07.229+0000 7f1f83c41380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:19:07.241 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:19:07.229+0000 7f1f83c41380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:19:07.241 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:19:07.229+0000 7f1f83c41380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:19:07.242 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:19:07.896 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:07.894548 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:19:08.597 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:08.594637 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:19:08.581+0000 7f81d4477380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:19:08.597 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:19:08.581+0000 7f81d4477380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:19:08.597 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:19:08.581+0000 7f81d4477380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:19:08.598 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:19:09.886 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:09.884339 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:19:10.554 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:10.552117 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:19:10.541+0000 7fa14085f380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:19:10.554 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:19:10.541+0000 7fa14085f380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:19:10.555 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:19:10.541+0000 7fa14085f380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:19:10.555 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:19:11.244 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:11.241963 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:19:11.256 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:11.254557 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:19:13.129 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:13.127129 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:19:13.803 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:13.800532 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:19:13.789+0000 7f03b7a23380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:19:13.803 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:19:13.789+0000 7f03b7a23380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:19:13.803 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:19:13.789+0000 7f03b7a23380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:19:13.803 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:19:18.933 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:18.931300 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:19:19.604 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:19.601661 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:19:19.589+0000 7ff868984380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:19:19.604 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:19:19.589+0000 7ff868984380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:19:19.604 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:19:19.589+0000 7ff868984380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:19:19.604 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:19:21.325 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:21.323913 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:19:21.336 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:21.334712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:19:29.852 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:29.850630 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:19:30.529 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:30.526952 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:19:30.513+0000 7f0849f81380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:19:30.529 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:19:30.513+0000 7f0849f81380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:19:30.530 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:19:30.513+0000 7f0849f81380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:19:30.530 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:19:31.414 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:31.412082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:19:31.426 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:31.425049 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:19:41.501 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:41.498993 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:19:41.513 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:41.511712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:19:51.017 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:51.014819 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:19:51.631 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:51.629723 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:19:51.644 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:51.642437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:19:51.675 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:19:51.672942 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:19:51.660+0000 7efc37dd6380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:19:51.675 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:19:51.660+0000 7efc37dd6380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:19:51.675 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:19:51.660+0000 7efc37dd6380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:19:51.675 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:20:01.787 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:20:01.785811 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:20:01.798 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:20:01.796821 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:20:11.921 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:20:11.919718 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:20:11.938 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:20:11.934423 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:20:22.051 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:20:22.049507 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:20:22.066 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:20:22.065282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:20:32.137 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:20:32.136112 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:20:32.150 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:20:32.147980 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:20:32.640 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:20:32.638528 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:20:33.287 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:20:33.284566 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:20:33.272+0000 7f9c2a5a9380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:20:33.287 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:20:33.272+0000 7f9c2a5a9380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:20:33.287 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:20:33.272+0000 7f9c2a5a9380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:20:33.287 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:20:42.231 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:20:42.229656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:20:42.246 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:20:42.244633 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:20:52.348 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:20:52.346007 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:20:52.359 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:20:52.358080 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:21:02.439 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:21:02.437564 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:21:02.485 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:21:02.483432 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:21:12.593 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:21:12.590993 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:21:12.607 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:21:12.606381 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:21:22.718 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:21:22.716790 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:21:22.731 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:21:22.729827 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:21:32.817 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:21:32.815064 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:21:32.830 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:21:32.828607 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:21:42.909 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:21:42.907098 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:21:42.920 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:21:42.919246 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:21:53.057 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:21:53.055169 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:21:53.068 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:21:53.066582 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:21:55.217 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:21:55.215208 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:21:55.926 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:21:55.924482 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:21:55.915+0000 7fbad9528380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:21:55.927 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:21:55.915+0000 7fbad9528380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:21:55.927 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:21:55.915+0000 7fbad9528380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:21:55.927 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:22:03.204 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:22:03.202808 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:22:03.217 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:22:03.216007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:22:13.373 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:22:13.369495 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:22:13.387 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:22:13.385809 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:22:23.529 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:22:23.527550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:22:23.543 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:22:23.542167 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:22:33.674 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:22:33.672911 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:22:33.685 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:22:33.683774 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:22:43.772 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:22:43.770279 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:22:43.787 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:22:43.785936 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:22:53.894 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:22:53.892235 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:22:53.910 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:22:53.908719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:23:03.996 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:23:03.994903 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:23:04.010 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:23:04.009195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:23:14.127 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:23:14.125809 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:23:14.140 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:23:14.138718 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:23:24.256 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:23:24.254866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:23:24.269 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:23:24.268336 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:23:34.339 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:23:34.337629 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:23:34.353 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:23:34.351635 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:23:44.445 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:23:44.443358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:23:44.458 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:23:44.456401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:23:54.525 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:23:54.523951 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:23:54.539 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:23:54.537696 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:24:04.649 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:24:04.647396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:24:04.664 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:24:04.663206 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:24:14.731 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:24:14.729833 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:24:14.744 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:24:14.743445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:24:24.846 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:24:24.844424 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:24:24.858 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:24:24.856808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:24:34.951 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:24:34.948613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:24:34.961 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:24:34.960030 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:24:39.774 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:24:39.772238 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:24:40.506 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:24:40.504592 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:24:40.492+0000 7f048aa1e380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:24:40.507 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:24:40.492+0000 7f048aa1e380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:24:40.507 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:24:40.492+0000 7f048aa1e380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:24:40.507 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:24:45.085 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:24:45.083682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:24:45.098 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:24:45.097212 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:24:55.234 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:24:55.232294 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:24:55.247 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:24:55.245748 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:25:05.319 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:25:05.317888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:25:05.333 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:25:05.332261 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:25:15.448 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:25:15.446855 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:25:15.462 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:25:15.461057 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:25:25.532 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:25:25.530556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:25:25.545 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:25:25.543554 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:25:35.650 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:25:35.648170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:25:35.662 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:25:35.660644 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:25:45.830 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:25:45.828920 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:25:45.842 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:25:45.841568 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:25:55.941 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:25:55.939385 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:25:55.989 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:25:55.988034 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:26:06.104 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:26:06.102262 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:26:06.118 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:26:06.116401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:26:16.201 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:26:16.199265 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:26:16.211 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:26:16.210472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:26:26.305 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:26:26.304233 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:26:26.319 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:26:26.318211 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:26:36.385 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:26:36.384280 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:26:36.395 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:26:36.394118 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:26:46.524 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:26:46.522533 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:26:46.535 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:26:46.534467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:26:56.660 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:26:56.659021 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:26:56.672 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:26:56.671094 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:27:06.749 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:27:06.747414 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:27:06.762 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:27:06.761299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:27:16.859 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:27:16.857421 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:27:16.871 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:27:16.870080 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:27:26.942 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:27:26.940713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:27:26.997 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:27:26.995783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:27:37.067 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:27:37.065442 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:27:37.079 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:27:37.078140 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:27:47.193 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:27:47.191316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:27:47.205 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:27:47.203596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:27:57.348 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:27:57.346192 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:27:57.359 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:27:57.357900 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:28:07.432 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:28:07.430350 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:28:07.488 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:28:07.486551 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:28:17.601 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:28:17.599547 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:28:17.614 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:28:17.613304 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:28:27.678 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:28:27.676812 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:28:27.690 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:28:27.688750 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:28:37.767 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:28:37.766047 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:28:37.780 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:28:37.779048 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:28:47.859 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:28:47.857603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:28:47.875 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:28:47.874106 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:28:57.975 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:28:57.973810 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:28:57.985 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:28:57.983594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:29:08.095 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:29:08.093489 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:29:08.108 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:29:08.106636 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:29:18.237 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:29:18.235122 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:29:18.251 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:29:18.250085 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:29:28.327 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:29:28.326066 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:29:28.341 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:29:28.340142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:29:38.424 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:29:38.423158 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:29:38.440 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:29:38.438786 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:29:48.555 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:29:48.553948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:29:48.569 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:29:48.568108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:29:58.680 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:29:58.678341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:29:58.693 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:29:58.692087 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:30:08.194 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:30:08.191985 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:30:08.790 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:30:08.788316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:30:08.800 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:30:08.799337 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:30:08.941 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:30:08.939309 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:30:08.930+0000 7f4341460380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:30:08.942 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:30:08.930+0000 7f4341460380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:30:08.943 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:30:08.930+0000 7f4341460380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:30:08.943 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:30:18.949 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:30:18.946915 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:30:18.962 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:30:18.960597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:30:29.031 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:30:29.024235 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:30:29.038 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:30:29.036699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:30:39.149 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:30:39.147615 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:30:39.163 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:30:39.161448 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:30:49.307 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:30:49.305541 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:30:49.320 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:30:49.318760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:30:59.432 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:30:59.430761 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:30:59.443 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:30:59.441702 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:31:09.512 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:31:09.510632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:31:09.572 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:31:09.570472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:31:19.669 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:31:19.667147 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:31:19.680 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:31:19.679135 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:31:29.793 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:31:29.791927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:31:29.806 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:31:29.804659 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:31:39.905 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:31:39.903607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:31:39.918 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:31:39.916482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:31:49.999 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:31:49.997842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:31:50.012 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:31:50.011154 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:32:00.099 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:32:00.097174 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:32:00.111 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:32:00.109624 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:32:10.224 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:32:10.222147 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:32:10.237 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:32:10.235626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:32:20.329 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:32:20.327229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:32:20.342 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:32:20.340360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:32:30.416 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:32:30.414217 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:32:30.431 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:32:30.429506 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:32:40.518 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:32:40.515838 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:32:40.531 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:32:40.529391 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:32:50.627 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:32:50.625067 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:32:50.641 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:32:50.639544 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:33:00.721 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:33:00.718636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:33:00.749 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:33:00.747177 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:33:10.812 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:33:10.810535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:33:10.826 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:33:10.824460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:33:20.921 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:33:20.919653 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:33:20.947 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:33:20.945894 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:33:31.071 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:33:31.068883 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:33:31.084 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:33:31.082766 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:33:41.153 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:33:41.151681 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:33:41.208 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:33:41.206420 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:33:51.319 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:33:51.317135 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:33:51.331 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:33:51.329385 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:34:01.467 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:34:01.464884 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:34:01.478 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:34:01.476737 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:34:11.540 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:34:11.538009 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:34:11.553 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:34:11.551446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:34:21.623 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:34:21.621445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:34:21.688 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:34:21.686671 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:34:31.763 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:34:31.761837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:34:31.776 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:34:31.774640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:34:41.843 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:34:41.841398 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:34:41.860 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:34:41.858713 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:34:51.966 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:34:51.964239 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:34:51.979 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:34:51.977396 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:35:02.098 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:35:02.095823 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:35:02.111 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:35:02.109623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:35:12.236 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:35:12.233689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:35:12.253 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:35:12.251844 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:35:22.332 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:35:22.330332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:35:22.344 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:35:22.342600 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:35:32.412 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:35:32.410088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:35:32.514 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:35:32.512437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:35:42.598 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:35:42.596305 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:35:42.610 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:35:42.608784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:35:52.708 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:35:52.705877 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:35:52.722 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:35:52.720153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:36:02.810 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:36:02.808045 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:36:02.820 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:36:02.818977 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:36:12.936 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:36:12.934245 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:36:12.950 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:36:12.948354 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:36:23.067 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:36:23.065030 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:36:23.080 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:36:23.078343 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:36:33.166 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:36:33.163664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:36:33.180 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:36:33.177622 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:36:43.325 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:36:43.323304 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:36:43.343 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:36:43.341225 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:36:53.428 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:36:53.426572 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:36:53.488 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:36:53.486216 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:37:03.656 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:37:03.654537 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:37:03.668 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:37:03.667101 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:37:13.758 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:37:13.755948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:37:13.816 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:37:13.814115 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:37:23.899 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:37:23.897601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:37:23.914 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:37:23.912285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:37:33.987 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:37:33.985028 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:37:34.000 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:37:33.998157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:37:44.109 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:37:44.107667 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:37:44.120 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:37:44.119005 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:37:54.206 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:37:54.203784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:37:54.217 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:37:54.216060 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:38:04.341 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:38:04.338942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:38:04.355 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:38:04.353851 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:38:14.442 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:38:14.440725 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:38:14.456 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:38:14.454939 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:38:24.577 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:38:24.575548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:38:24.593 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:38:24.591786 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:38:34.706 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:38:34.704056 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:38:34.721 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:38:34.719610 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:38:44.840 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:38:44.838212 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:38:44.855 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:38:44.853837 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:38:54.985 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:38:54.983323 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:38:54.998 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:38:54.996554 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:39:05.071 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:39:05.069847 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:39:05.083 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:39:05.082126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:39:15.279 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:39:15.276926 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:39:15.290 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:39:15.287990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:39:25.432 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:39:25.430433 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:39:25.447 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:39:25.445422 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:39:35.541 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:39:35.539727 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:39:35.556 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:39:35.553920 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:39:45.646 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:39:45.643995 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:39:45.738 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:39:45.735962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:39:55.839 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:39:55.837043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:39:55.851 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:39:55.849201 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:40:05.988 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:40:05.986288 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:40:06.002 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:40:06.000262 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:40:16.202 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:40:16.200759 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:40:16.212 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:40:16.210956 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:40:26.315 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:40:26.313744 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:40:26.326 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:40:26.324968 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:40:36.431 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:40:36.425754 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:40:36.440 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:40:36.438998 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:40:46.504 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:40:46.501771 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:40:46.521 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:40:46.520123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:40:56.642 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:40:56.640102 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:40:56.655 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:40:56.654123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:41:04.313 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:41:04.311678 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:41:05.003 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:41:05.001273 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:41:04.991+0000 7ff4c8f56380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:41:05.004 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:41:04.991+0000 7ff4c8f56380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:41:05.004 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:41:04.991+0000 7ff4c8f56380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:41:05.004 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:41:06.722 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:41:06.719956 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:41:06.732 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:41:06.731075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:41:16.836 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:41:16.834119 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:41:16.867 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:41:16.865995 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:41:27.005 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:41:27.003273 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:41:27.026 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:41:27.024604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:41:37.098 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:41:37.095945 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:41:37.112 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:41:37.110623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:41:47.197 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:41:47.195208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:41:47.211 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:41:47.209850 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:41:57.299 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:41:57.297374 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:41:57.312 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:41:57.310900 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:42:07.409 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:42:07.407080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:42:07.423 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:42:07.421686 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:42:17.503 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:42:17.501097 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:42:17.520 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:42:17.518485 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:42:27.714 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:42:27.712024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:42:27.724 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:42:27.722843 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:42:37.899 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:42:37.896822 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:42:37.950 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:42:37.948232 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:42:48.023 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:42:48.021715 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:42:48.037 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:42:48.036062 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:42:58.150 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:42:58.148314 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:42:58.162 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:42:58.160524 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:43:08.231 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:43:08.229044 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:43:08.244 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:43:08.242306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:43:18.455 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:43:18.453186 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:43:18.466 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:43:18.464756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:43:28.573 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:43:28.571457 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:43:28.587 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:43:28.586044 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:43:38.690 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:43:38.688731 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:43:38.705 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:43:38.704017 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:43:48.809 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:43:48.807046 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:43:48.821 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:43:48.819480 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:43:58.938 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:43:58.936697 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:43:58.952 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:43:58.950415 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:44:09.023 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:44:09.021416 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:44:09.122 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:44:09.120365 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:44:19.248 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:44:19.246744 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:44:19.261 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:44:19.259316 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:44:29.372 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:44:29.370913 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:44:29.384 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:44:29.382868 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:44:39.450 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:44:39.447886 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:44:39.466 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:44:39.464156 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:44:49.644 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:44:49.642395 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:44:49.654 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:44:49.652982 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:44:59.762 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:44:59.760196 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:44:59.771 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:44:59.770161 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:45:09.838 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:45:09.836009 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:45:09.851 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:45:09.849852 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:45:19.958 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:45:19.956203 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:45:19.971 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:45:19.969841 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:45:30.102 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:45:30.099902 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:45:30.117 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:45:30.115283 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:45:40.288 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:45:40.286248 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:45:40.302 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:45:40.300928 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:45:50.403 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:45:50.401437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:45:50.483 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:45:50.481179 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:46:00.595 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:46:00.593226 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:46:00.608 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:46:00.606742 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:46:10.685 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:46:10.682798 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:46:10.697 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:46:10.695308 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:46:20.800 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:46:20.798558 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:46:20.814 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:46:20.812600 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:46:30.922 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:46:30.920056 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:46:30.936 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:46:30.934840 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:46:41.074 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:46:41.072030 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:46:41.087 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:46:41.085911 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:46:51.184 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:46:51.182168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:46:51.197 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:46:51.195129 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:47:01.264 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:47:01.262684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:47:01.317 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:47:01.315224 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:47:11.397 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:47:11.395577 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:47:11.916 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:47:11.407003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:47:21.523 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:47:21.521168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:47:21.537 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:47:21.535480 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:47:31.606 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:47:31.604270 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:47:31.620 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:47:31.618406 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:47:41.717 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:47:41.715448 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:47:41.732 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:47:41.730663 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:47:51.877 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:47:51.874651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:47:51.891 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:47:51.887484 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:48:02.000 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:48:01.997835 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:48:02.013 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:48:02.011739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:48:12.078 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:48:12.076287 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:48:12.091 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:48:12.089954 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:48:22.226 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:48:22.223427 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:48:22.239 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:48:22.237819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:48:32.305 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:48:32.303293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:48:32.318 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:48:32.316268 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:48:42.396 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:48:42.393802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:48:42.422 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:48:42.416988 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:48:52.532 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:48:52.530695 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:48:52.543 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:48:52.541620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:49:02.612 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:49:02.609889 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:49:02.627 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:49:02.625321 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:49:12.835 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:49:12.833445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:49:12.846 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:49:12.843987 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:49:22.923 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:49:22.920724 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:49:22.982 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:49:22.980547 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:49:33.097 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:49:33.095156 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:49:33.113 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:49:33.111660 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:49:43.216 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:49:43.214226 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:49:43.229 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:49:43.227340 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:49:53.356 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:49:53.354050 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:49:53.367 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:49:53.365336 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:50:03.487 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:50:03.485073 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:50:03.499 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:50:03.497235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:50:13.600 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:50:13.597997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:50:13.613 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:50:13.611957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:50:23.698 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:50:23.696086 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:50:23.710 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:50:23.708374 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:50:33.854 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:50:33.852205 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:50:33.867 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:50:33.865824 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:50:43.961 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:50:43.958656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:50:43.976 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:50:43.974654 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:50:54.046 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:50:54.044207 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:50:54.061 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:50:54.059020 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:51:04.252 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:51:04.250747 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:51:04.268 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:51:04.266194 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:51:14.357 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:51:14.354845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:51:14.371 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:51:14.369184 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:51:24.515 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:51:24.512888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:51:24.528 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:51:24.526590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:51:34.636 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:51:34.633436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:51:34.649 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:51:34.647299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:51:44.725 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:51:44.722863 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:51:44.774 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:51:44.772893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:51:54.880 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:51:54.878185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:51:54.897 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:51:54.894585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:52:05.024 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:52:05.021947 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:52:05.032 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:52:05.031039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:52:15.109 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:52:15.107082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:52:15.134 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:52:15.120174 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:52:25.222 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:52:25.220303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:52:25.236 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:52:25.234177 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:52:35.327 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:52:35.325448 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:52:35.341 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:52:35.339082 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:52:45.425 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:52:45.422556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:52:45.440 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:52:45.438191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:52:55.545 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:52:55.542515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:52:55.559 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:52:55.557669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:53:05.647 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:53:05.644753 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:53:05.659 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:53:05.657379 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:53:15.726 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:53:15.723745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:53:15.739 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:53:15.737087 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:53:25.854 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:53:25.852202 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:53:25.870 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:53:25.868050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:53:36.013 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:53:36.010600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:53:36.023 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:53:36.021644 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:53:46.163 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:53:46.161063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:53:46.176 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:53:46.174194 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:53:56.460 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:53:56.457665 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:53:56.473 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:53:56.470892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:54:06.604 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:54:06.602180 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:54:06.616 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:54:06.614772 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:54:16.739 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:54:16.737666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:54:16.754 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:54:16.752817 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:54:26.828 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:54:26.826631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:54:26.838 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:54:26.836549 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:54:36.907 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:54:36.904862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:54:36.962 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:54:36.960484 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:54:47.027 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:54:47.025608 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:54:47.043 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:54:47.041111 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:54:57.163 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:54:57.160966 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:54:57.174 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:54:57.172544 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:55:07.320 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:55:07.318140 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:55:07.336 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:55:07.334806 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:55:17.458 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:55:17.456132 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:55:17.472 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:55:17.470512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:55:27.543 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:55:27.541466 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:55:27.559 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:55:27.556698 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:55:37.687 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:55:37.685389 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:55:37.699 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:55:37.697144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:55:47.821 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:55:47.819097 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:55:47.833 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:55:47.831099 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:55:57.946 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:55:57.943934 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:55:57.966 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:55:57.964222 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:56:08.069 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:56:08.067518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:56:08.080 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:56:08.078744 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:56:18.203 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:56:18.201323 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:56:18.216 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:56:18.214458 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:56:28.342 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:56:28.339610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:56:28.360 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:56:28.355350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:56:38.525 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:56:38.523085 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:56:38.537 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:56:38.535961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:56:48.634 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:56:48.631854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:56:48.649 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:56:48.648031 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:56:58.770 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:56:58.767638 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:56:58.784 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:56:58.782589 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:57:08.889 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:57:08.886881 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:57:08.903 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:57:08.901579 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:57:18.980 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:57:18.978467 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:57:18.993 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:57:18.991728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:57:29.106 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:57:29.103582 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:57:29.119 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:57:29.117701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:57:39.220 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:57:39.218508 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:57:39.234 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:57:39.232245 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:57:45.010 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:57:45.008176 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T15:57:45.696 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:57:45.693421 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T15:57:45.683+0000 7ff4441bf380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T15:57:45.696 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:57:45.683+0000 7ff4441bf380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T15:57:45.696 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T15:57:45.683+0000 7ff4441bf380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T15:57:45.696 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T15:57:49.301 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:57:49.299036 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:57:49.366 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:57:49.363696 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:57:59.439 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:57:59.437267 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:57:59.452 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:57:59.449845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:58:09.528 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:58:09.526426 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:58:09.644 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:58:09.642049 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:58:19.762 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:58:19.760337 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:58:19.776 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:58:19.774150 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:58:29.840 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:58:29.838896 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:58:29.890 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:58:29.888224 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:58:39.995 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:58:39.992832 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:58:40.009 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:58:40.007897 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:58:50.143 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:58:50.141932 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:58:50.156 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:58:50.154976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:59:00.273 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:59:00.270715 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:59:00.286 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:59:00.284354 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:59:10.362 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:59:10.360248 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:59:10.375 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:59:10.373550 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:59:20.472 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:59:20.470150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:59:20.485 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:59:20.483129 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:59:30.559 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:59:30.557146 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:59:30.571 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:59:30.569515 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:59:40.664 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:59:40.662147 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:59:40.679 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:59:40.677457 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:59:50.777 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:59:50.775318 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T15:59:50.794 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 15:59:50.792058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:00:00.855 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:00:00.851682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:00:00.865 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:00:00.863296 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:00:10.960 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:00:10.958498 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:00:10.973 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:00:10.971995 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:00:21.073 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:00:21.070968 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:00:21.085 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:00:21.083597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:00:31.211 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:00:31.209440 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:00:31.220 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:00:31.218841 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:00:41.317 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:00:41.315061 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:00:41.332 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:00:41.330544 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:00:51.404 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:00:51.401945 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:00:51.486 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:00:51.484091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:01:01.572 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:01:01.570093 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:01:01.587 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:01:01.585717 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:01:11.656 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:01:11.654185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:01:11.679 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:01:11.676959 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:01:21.754 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:01:21.752204 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:01:21.770 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:01:21.767749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:01:31.899 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:01:31.897222 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:01:31.912 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:01:31.910614 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:01:42.056 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:01:42.054612 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:01:42.239 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:01:42.237962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:01:52.311 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:01:52.309181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:01:52.321 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:01:52.320300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:02:02.389 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:02:02.387402 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:02:02.402 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:02:02.401165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:02:12.517 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:02:12.514929 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:02:12.529 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:02:12.527932 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:02:22.649 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:02:22.647383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:02:22.662 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:02:22.661057 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:02:32.802 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:02:32.800175 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:02:32.814 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:02:32.812631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:02:42.885 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:02:42.883364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:02:42.902 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:02:42.900728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:02:53.011 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:02:53.009477 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:02:53.025 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:02:53.023373 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:03:03.125 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:03:03.122906 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:03:03.136 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:03:03.134658 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:03:13.214 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:03:13.212246 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:03:13.228 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:03:13.226963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:03:23.304 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:03:23.302321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:03:23.317 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:03:23.315914 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:03:33.415 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:03:33.413453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:03:33.431 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:03:33.429345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:03:43.510 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:03:43.507943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:03:43.523 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:03:43.521959 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:03:53.662 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:03:53.660659 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:03:53.677 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:03:53.675347 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:04:03.789 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:04:03.787116 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:04:03.802 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:04:03.800497 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:04:13.904 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:04:13.902550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:04:13.920 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:04:13.918726 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:04:24.036 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:04:24.034419 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:04:24.050 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:04:24.048292 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:04:34.179 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:04:34.177510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:04:34.193 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:04:34.192025 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:04:44.265 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:04:44.263616 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:04:44.280 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:04:44.278308 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:04:54.379 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:04:54.377807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:04:54.393 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:04:54.391454 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:05:04.493 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:05:04.491580 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:05:04.507 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:05:04.505357 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:05:14.618 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:05:14.616302 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:05:14.631 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:05:14.629629 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:05:24.734 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:05:24.732368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:05:24.746 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:05:24.744763 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:05:34.855 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:05:34.853393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:05:34.867 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:05:34.865246 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:05:44.991 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:05:44.989786 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:05:45.007 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:05:45.005603 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:05:55.101 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:05:55.099340 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:05:55.126 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:05:55.120881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:06:05.274 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:06:05.271907 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:06:05.290 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:06:05.288440 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:06:15.405 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:06:15.403627 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:06:15.415 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:06:15.414299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:06:25.525 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:06:25.523516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:06:25.539 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:06:25.537786 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:06:35.604 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:06:35.602445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:06:35.616 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:06:35.614921 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:06:45.774 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:06:45.772253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:06:45.785 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:06:45.783724 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:06:55.925 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:06:55.923137 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:06:55.941 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:06:55.935188 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:07:06.026 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:07:06.024669 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:07:06.040 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:07:06.039085 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:07:16.106 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:07:16.104210 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:07:16.120 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:07:16.118728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:07:26.191 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:07:26.189302 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:07:26.204 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:07:26.202527 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:07:36.291 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:07:36.289769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:07:36.348 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:07:36.346349 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:07:46.467 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:07:46.465040 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:07:46.479 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:07:46.477582 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:07:56.590 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:07:56.587986 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:07:56.604 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:07:56.602796 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:08:06.709 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:08:06.708095 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:08:06.723 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:08:06.721563 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:08:16.846 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:08:16.844619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:08:16.860 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:08:16.859212 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:08:26.933 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:08:26.929142 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:08:26.991 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:08:26.990410 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:08:37.104 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:08:37.102430 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:08:37.114 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:08:37.112608 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:08:47.187 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:08:47.185698 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:08:47.254 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:08:47.252251 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:08:57.539 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:08:57.537510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:08:57.553 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:08:57.551974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:09:07.630 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:09:07.628956 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:09:07.675 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:09:07.673817 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:09:17.807 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:09:17.805357 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:09:17.819 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:09:17.818290 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:09:27.923 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:09:27.922127 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:09:27.939 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:09:27.938099 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:09:38.073 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:09:38.071504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:09:38.087 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:09:38.086294 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:09:48.213 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:09:48.211721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:09:48.226 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:09:48.225407 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:09:58.344 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:09:58.342522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:09:58.359 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:09:58.358139 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:10:08.466 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:10:08.464207 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:10:08.483 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:10:08.481819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:10:18.620 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:10:18.618476 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:10:18.651 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:10:18.649964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:10:28.727 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:10:28.725769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:10:28.743 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:10:28.742032 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:10:38.808 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:10:38.806724 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:10:38.865 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:10:38.864367 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:10:48.971 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:10:48.969342 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:10:48.982 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:10:48.981099 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:10:59.074 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:10:59.072693 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:10:59.090 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:10:59.089565 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:11:09.191 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:11:09.190339 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:11:09.204 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:11:09.203472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:11:19.306 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:11:19.304501 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:11:19.319 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:11:19.318598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:11:29.398 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:11:29.396869 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:11:29.410 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:11:29.409103 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:11:39.481 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:11:39.480266 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:11:39.496 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:11:39.494856 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:11:49.617 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:11:49.615660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:11:49.629 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:11:49.628633 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:11:59.725 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:11:59.723542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:11:59.737 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:11:59.736798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:12:09.839 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:12:09.838343 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:12:09.852 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:12:09.851210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:12:19.960 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:12:19.959087 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:12:19.973 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:12:19.972488 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:12:30.069 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:12:30.068146 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:12:30.083 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:12:30.082668 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:12:40.270 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:12:40.269223 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:12:40.284 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:12:40.283587 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:12:50.355 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:12:50.354128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:12:50.368 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:12:50.367090 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:13:00.472 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:13:00.471057 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:13:00.486 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:13:00.485152 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:13:10.609 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:13:10.607690 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:13:10.621 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:13:10.620384 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:13:20.721 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:13:20.720434 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:13:20.736 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:13:20.735047 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:13:30.813 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:13:30.811625 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:13:30.823 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:13:30.822598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:13:40.897 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:13:40.896237 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:13:40.953 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:13:40.952385 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:13:51.022 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:13:51.021441 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:13:51.034 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:13:51.033609 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:14:01.133 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:14:01.131669 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:14:01.145 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:14:01.144824 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:14:11.250 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:14:11.248820 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:14:11.262 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:14:11.261388 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:14:21.333 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:14:21.332398 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:14:21.346 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:14:21.345423 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:14:25.701 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:14:25.699654 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T16:14:26.447 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:14:26.445518 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T16:14:26.435+0000 7faaa68a8380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T16:14:26.447 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T16:14:26.435+0000 7faaa68a8380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T16:14:26.447 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T16:14:26.435+0000 7faaa68a8380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T16:14:26.448 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T16:14:31.418 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:14:31.417154 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:14:31.430 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:14:31.429191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:14:41.562 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:14:41.560882 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:14:41.575 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:14:41.574166 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:14:51.662 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:14:51.661135 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:14:51.696 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:14:51.695654 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:15:01.830 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:15:01.828764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:15:01.844 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:15:01.843072 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:15:12.026 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:15:12.025042 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:15:12.040 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:15:12.039193 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:15:22.172 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:15:22.170881 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:15:22.186 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:15:22.185479 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:15:32.273 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:15:32.272194 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:15:32.322 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:15:32.320834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:15:42.415 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:15:42.413820 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:15:42.430 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:15:42.429215 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:15:52.564 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:15:52.562604 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:15:52.575 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:15:52.574044 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:16:02.668 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:16:02.667673 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:16:02.682 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:16:02.681310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:16:12.759 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:16:12.758423 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:16:12.771 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:16:12.770784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:16:22.896 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:16:22.894932 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:16:22.908 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:16:22.907775 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:16:32.973 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:16:32.971930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:16:32.984 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:16:32.984039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:16:43.073 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:16:43.072180 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:16:43.085 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:16:43.084654 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:16:53.158 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:16:53.157453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:16:53.170 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:16:53.169133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:17:03.291 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:17:03.289745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:17:03.304 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:17:03.303399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:17:13.376 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:17:13.375617 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:17:13.392 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:17:13.391402 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:17:23.460 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:17:23.458754 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:17:23.508 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:17:23.506882 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:17:33.641 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:17:33.640128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:17:33.655 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:17:33.653957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:17:43.771 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:17:43.764092 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:17:43.783 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:17:43.781620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:17:53.882 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:17:53.880955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:17:53.895 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:17:53.893654 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:18:03.975 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:18:03.974317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:18:03.987 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:18:03.986638 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:18:14.061 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:18:14.060204 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:18:14.074 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:18:14.074043 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:18:24.142 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:18:24.141414 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:18:24.154 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:18:24.153814 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:18:34.224 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:18:34.223110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:18:34.241 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:18:34.240161 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:18:44.363 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:18:44.362642 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:18:44.375 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:18:44.374066 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:18:54.450 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:18:54.449490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:18:54.499 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:18:54.498368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:19:04.570 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:19:04.568491 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:19:04.582 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:19:04.581820 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:19:14.657 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:19:14.656224 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:19:14.671 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:19:14.670379 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:19:24.745 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:19:24.744137 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:19:24.767 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:19:24.766130 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:19:34.847 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:19:34.846732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:19:34.883 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:19:34.882100 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:19:44.951 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:19:44.950547 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:19:44.964 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:19:44.964068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:19:55.037 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:19:55.036461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:19:55.050 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:19:55.049766 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:20:05.114 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:20:05.113247 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:20:05.126 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:20:05.125228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:20:15.235 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:20:15.234006 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:20:15.246 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:20:15.245583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:20:25.402 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:20:25.400719 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:20:25.412 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:20:25.411396 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:20:35.500 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:20:35.499341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:20:35.515 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:20:35.514655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:20:45.585 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:20:45.584447 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:20:45.601 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:20:45.600479 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:20:55.719 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:20:55.718226 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:20:55.731 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:20:55.730246 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:21:05.842 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:21:05.840740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:21:05.856 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:21:05.855262 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:21:15.973 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:21:15.972432 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:21:15.990 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:21:15.989601 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:21:26.215 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:21:26.214408 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:21:26.228 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:21:26.228036 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:21:36.348 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:21:36.347930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:21:36.361 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:21:36.360381 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:21:46.463 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:21:46.462273 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:21:46.479 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:21:46.478285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:21:56.572 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:21:56.570838 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:21:56.582 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:21:56.581768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:22:06.651 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:22:06.650302 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:22:06.664 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:22:06.663512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:22:16.760 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:22:16.759786 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:22:16.774 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:22:16.773179 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:22:26.955 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:22:26.954901 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:22:26.968 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:22:26.967421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:22:37.087 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:22:37.085822 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:22:37.099 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:22:37.099129 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:22:47.220 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:22:47.219705 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:22:47.234 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:22:47.233491 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:22:57.311 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:22:57.310639 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:22:57.327 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:22:57.326471 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:23:07.446 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:23:07.445088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:23:07.459 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:23:07.458454 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:23:17.524 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:23:17.522867 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:23:17.537 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:23:17.536648 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:23:27.618 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:23:27.617641 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:23:27.710 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:23:27.708996 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:23:37.802 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:23:37.801213 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:23:37.815 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:23:37.814318 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:23:47.892 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:23:47.890664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:23:47.905 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:23:47.904017 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:23:57.980 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:23:57.978839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:23:57.996 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:23:57.995445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:24:08.110 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:24:08.109609 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:24:08.122 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:24:08.121212 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:24:18.190 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:24:18.189315 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:24:18.205 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:24:18.204198 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:24:28.316 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:24:28.315546 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:24:28.329 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:24:28.328039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:24:38.471 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:24:38.469584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:24:38.486 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:24:38.485743 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:24:48.557 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:24:48.556871 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:24:48.606 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:24:48.605413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:24:58.706 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:24:58.705445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:24:58.720 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:24:58.719533 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:25:08.848 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:25:08.847403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:25:08.861 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:25:08.860900 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:25:18.998 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:25:18.997047 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:25:19.012 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:25:19.011596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:25:29.191 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:25:29.190125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:25:29.242 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:25:29.241256 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:25:39.369 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:25:39.368439 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:25:39.382 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:25:39.381649 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:25:49.462 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:25:49.460982 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:25:49.476 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:25:49.475413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:25:59.547 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:25:59.545846 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:25:59.560 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:25:59.559449 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:26:09.634 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:26:09.632967 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:26:09.645 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:26:09.644891 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:26:19.768 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:26:19.767142 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:26:19.783 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:26:19.782063 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:26:29.869 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:26:29.868219 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:26:29.885 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:26:29.883469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:26:39.953 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:26:39.952162 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:26:39.965 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:26:39.964583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:26:50.040 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:26:50.038859 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:26:50.104 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:26:50.103531 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:27:00.183 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:27:00.182350 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:27:00.197 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:27:00.196690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:27:10.263 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:27:10.262015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:27:10.312 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:27:10.311114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:27:20.413 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:27:20.412275 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:27:20.423 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:27:20.422148 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:27:30.492 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:27:30.491002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:27:30.525 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:27:30.502959 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:27:40.577 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:27:40.576447 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:27:40.593 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:27:40.592183 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:27:50.699 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:27:50.698013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:27:50.712 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:27:50.711406 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:28:00.844 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:28:00.842811 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:28:00.857 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:28:00.855988 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:28:10.936 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:28:10.935503 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:28:10.947 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:28:10.947030 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:28:21.043 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:28:21.042103 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:28:21.056 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:28:21.055788 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:28:31.169 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:28:31.168082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:28:31.178 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:28:31.177594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:28:41.291 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:28:41.289796 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:28:41.303 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:28:41.302785 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:28:51.379 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:28:51.377860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:28:51.389 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:28:51.388961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:29:01.561 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:29:01.559925 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:29:01.572 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:29:01.571902 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:29:11.683 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:29:11.681953 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:29:11.695 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:29:11.694027 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:29:21.759 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:29:21.757733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:29:21.801 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:29:21.800171 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:29:31.883 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:29:31.882534 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:29:31.897 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:29:31.896673 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:29:42.053 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:29:42.052035 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:29:42.067 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:29:42.066266 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:29:52.174 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:29:52.172578 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:29:52.204 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:29:52.203151 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:30:02.268 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:30:02.267181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:30:02.281 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:30:02.280354 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:30:12.394 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:30:12.393170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:30:12.408 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:30:12.407218 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:30:22.522 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:30:22.520965 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:30:22.533 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:30:22.532503 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:30:32.640 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:30:32.639470 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:30:32.653 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:30:32.652656 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:30:42.755 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:30:42.754184 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:30:42.769 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:30:42.767931 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:30:52.840 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:30:52.839243 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:30:52.855 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:30:52.854110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:31:02.965 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:31:02.963844 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:31:02.977 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:31:02.977055 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:31:06.461 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:31:06.459721 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T16:31:07.156 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:31:07.154535 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T16:31:07.147+0000 7fc56f131380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T16:31:07.157 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T16:31:07.147+0000 7fc56f131380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T16:31:07.157 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T16:31:07.147+0000 7fc56f131380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T16:31:07.157 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T16:31:13.086 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:31:13.084682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:31:13.100 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:31:13.099527 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:31:23.193 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:31:23.191931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:31:23.207 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:31:23.206661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:31:33.286 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:31:33.285132 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:31:33.299 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:31:33.298102 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:31:43.409 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:31:43.407944 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:31:43.422 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:31:43.421643 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:31:53.529 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:31:53.527622 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:31:53.543 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:31:53.542616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:32:03.617 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:32:03.616560 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:32:03.630 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:32:03.629303 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:32:13.696 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:32:13.695026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:32:13.709 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:32:13.708436 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:32:23.817 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:32:23.816213 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:32:23.830 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:32:23.829216 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:32:33.942 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:32:33.940774 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:32:33.957 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:32:33.956258 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:32:44.096 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:32:44.094721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:32:44.108 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:32:44.106859 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:32:54.204 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:32:54.203497 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:32:54.219 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:32:54.218210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:33:04.378 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:33:04.374793 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:33:04.389 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:33:04.388387 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:33:14.468 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:33:14.467675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:33:14.482 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:33:14.481497 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:33:24.586 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:33:24.584964 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:33:24.600 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:33:24.599182 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:33:34.705 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:33:34.703825 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:33:34.721 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:33:34.720259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:33:44.796 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:33:44.795276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:33:44.813 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:33:44.812707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:33:54.883 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:33:54.882483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:33:54.896 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:33:54.895024 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:34:04.987 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:34:04.986204 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:34:05.004 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:34:05.003301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:34:15.107 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:34:15.106269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:34:15.119 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:34:15.118089 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:34:25.203 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:34:25.202512 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:34:25.222 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:34:25.216696 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:34:35.331 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:34:35.329694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:34:35.351 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:34:35.350306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:34:45.426 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:34:45.425229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:34:45.438 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:34:45.437953 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:34:55.552 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:34:55.550842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:34:55.564 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:34:55.563718 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:35:05.652 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:35:05.650586 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:35:05.721 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:35:05.720720 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:35:15.825 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:35:15.824213 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:35:15.839 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:35:15.836301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:35:25.904 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:35:25.900426 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:35:25.914 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:35:25.913807 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:35:35.983 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:35:35.981757 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:35:36.044 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:35:36.041236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:35:46.149 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:35:46.148020 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:35:46.162 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:35:46.161620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:35:56.247 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:35:56.245602 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:35:56.258 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:35:56.257092 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:36:06.367 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:36:06.366049 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:36:06.381 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:36:06.380112 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:36:16.452 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:36:16.451230 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:36:16.465 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:36:16.464547 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:36:26.559 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:36:26.558490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:36:26.580 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:36:26.578964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:36:36.653 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:36:36.652247 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:36:36.666 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:36:36.665378 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:36:46.768 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:36:46.761890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:36:46.777 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:36:46.776038 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:36:56.860 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:36:56.859106 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:36:56.919 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:36:56.918599 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:37:07.029 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:37:07.027836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:37:07.039 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:37:07.038726 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:37:17.162 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:37:17.161688 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:37:17.177 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:37:17.176386 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:37:27.284 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:37:27.282979 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:37:27.297 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:37:27.296597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:37:37.419 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:37:37.418112 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:37:37.435 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:37:37.434756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:37:47.533 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:37:47.532521 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:37:47.546 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:37:47.545879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:37:57.700 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:37:57.699438 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:37:57.716 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:37:57.715128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:38:07.849 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:38:07.848164 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:38:07.862 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:38:07.861546 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:38:18.051 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:38:18.049609 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:38:18.060 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:38:18.058995 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:38:28.183 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:38:28.182215 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:38:28.195 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:38:28.194884 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:38:38.276 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:38:38.274918 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:38:38.290 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:38:38.289727 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:38:48.359 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:38:48.358646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:38:48.372 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:38:48.371197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:38:58.479 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:38:58.478173 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:38:58.494 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:38:58.493437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:39:08.576 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:39:08.574535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:39:08.588 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:39:08.587865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:39:18.687 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:39:18.686299 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:39:18.700 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:39:18.698709 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:39:28.779 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:39:28.778361 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:39:28.793 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:39:28.792163 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:39:38.891 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:39:38.890054 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:39:38.902 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:39:38.901484 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:39:49.040 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:39:49.038743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:39:49.055 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:39:49.054269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:39:59.146 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:39:59.144941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:39:59.157 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:39:59.156664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:40:09.273 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:40:09.272176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:40:09.288 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:40:09.287945 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:40:19.384 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:40:19.383124 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:40:19.398 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:40:19.397416 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:40:29.512 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:40:29.511243 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:40:29.525 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:40:29.524580 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:40:39.653 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:40:39.652125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:40:39.665 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:40:39.664665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:40:49.734 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:40:49.733177 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:40:49.799 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:40:49.797730 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:40:59.896 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:40:59.895022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:40:59.911 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:40:59.910700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:41:10.005 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:41:10.003504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:41:10.020 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:41:10.019395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:41:20.115 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:41:20.114617 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:41:20.133 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:41:20.130715 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:41:30.254 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:41:30.252984 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:41:30.265 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:41:30.264247 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:41:40.320 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:41:40.318709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:41:40.332 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:41:40.331564 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:41:50.403 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:41:50.402430 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:41:50.416 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:41:50.415336 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:42:00.480 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:42:00.478818 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:42:00.492 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:42:00.491388 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:42:10.578 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:42:10.576723 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:42:10.593 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:42:10.591630 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:42:20.715 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:42:20.714398 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:42:20.748 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:42:20.747472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:42:30.889 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:42:30.888475 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:42:30.902 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:42:30.901058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:42:41.053 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:42:41.051766 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:42:41.063 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:42:41.062196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:42:51.137 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:42:51.136167 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:42:51.204 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:42:51.202440 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:43:01.311 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:43:01.310193 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:43:01.326 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:43:01.325655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:43:11.401 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:43:11.399456 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:43:11.414 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:43:11.413156 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:43:21.531 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:43:21.530336 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:43:21.545 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:43:21.544172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:43:31.645 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:43:31.643889 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:43:31.656 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:43:31.655535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:43:41.763 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:43:41.761947 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:43:41.775 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:43:41.774916 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:43:51.896 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:43:51.894386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:43:51.912 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:43:51.911080 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:44:02.025 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:44:02.023999 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:44:02.041 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:44:02.040419 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:44:12.124 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:44:12.123221 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:44:12.138 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:44:12.137159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:44:22.273 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:44:22.272100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:44:22.287 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:44:22.285849 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:44:32.431 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:44:32.429782 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:44:32.444 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:44:32.443265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:44:42.591 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:44:42.590326 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:44:42.603 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:44:42.602407 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:44:52.721 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:44:52.719623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:44:52.731 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:44:52.730020 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:45:02.831 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:45:02.830194 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:45:02.852 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:45:02.848359 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:45:12.925 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:45:12.923788 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:45:12.939 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:45:12.938803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:45:23.017 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:45:23.015529 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:45:23.088 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:45:23.073298 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:45:33.154 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:45:33.152985 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:45:33.216 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:45:33.214633 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:45:43.327 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:45:43.326090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:45:43.340 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:45:43.339062 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:45:53.424 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:45:53.423062 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:45:53.435 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:45:53.434495 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:46:03.510 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:46:03.508629 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:46:03.524 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:46:03.523093 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:46:13.622 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:46:13.621026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:46:13.634 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:46:13.633639 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:46:23.708 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:46:23.706589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:46:23.722 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:46:23.721016 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:46:33.828 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:46:33.826943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:46:33.843 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:46:33.842625 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:46:43.963 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:46:43.961580 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:46:43.975 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:46:43.974423 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:46:54.086 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:46:54.085109 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:46:54.100 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:46:54.099772 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:47:04.235 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:47:04.234303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:47:04.246 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:47:04.245380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:47:14.323 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:47:14.321919 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:47:14.395 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:47:14.394245 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:47:24.492 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:47:24.490800 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:47:24.505 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:47:24.504215 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:47:34.586 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:47:34.584835 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:47:34.599 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:47:34.598701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:47:44.678 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:47:44.677504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:47:44.689 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:47:44.688609 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:47:47.162 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:47:47.161266 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T16:47:47.828 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:47:47.826489 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T16:47:47.818+0000 7f507b1c9380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T16:47:47.828 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T16:47:47.818+0000 7f507b1c9380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T16:47:47.829 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T16:47:47.818+0000 7f507b1c9380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T16:47:47.829 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T16:47:54.800 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:47:54.799074 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:47:54.814 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:47:54.813029 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:48:04.925 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:48:04.923418 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:48:04.941 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:48:04.940337 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:48:15.017 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:48:15.015594 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:48:15.028 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:48:15.027551 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:48:25.117 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:48:25.115825 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:48:25.146 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:48:25.145313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:48:35.324 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:48:35.322896 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:48:35.336 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:48:35.335097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:48:45.422 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:48:45.420963 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:48:45.436 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:48:45.435325 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:48:55.505 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:48:55.504155 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:48:55.519 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:48:55.518236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:49:05.620 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:49:05.618671 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:49:05.633 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:49:05.632317 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:49:15.729 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:49:15.728040 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:49:15.745 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:49:15.744686 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:49:25.847 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:49:25.846270 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:49:25.869 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:49:25.868229 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:49:35.952 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:49:35.951304 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:49:35.966 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:49:35.965624 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:49:46.110 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:49:46.109005 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:49:46.124 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:49:46.123103 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:49:56.206 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:49:56.204712 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:49:56.316 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:49:56.217517 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:50:06.329 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:50:06.327835 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:50:06.341 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:50:06.340937 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:50:16.438 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:50:16.437272 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:50:16.447 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:50:16.446209 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:50:26.553 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:50:26.551551 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:50:26.567 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:50:26.566718 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:50:36.651 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:50:36.649809 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:50:36.659 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:50:36.658753 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:50:46.735 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:50:46.734585 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:50:46.748 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:50:46.747279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:50:56.843 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:50:56.842162 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:50:56.855 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:50:56.854074 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:51:06.953 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:51:06.951627 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:51:06.966 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:51:06.965132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:51:17.057 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:51:17.056077 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:51:17.075 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:51:17.074188 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:51:27.182 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:51:27.181362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:51:27.196 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:51:27.195292 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:51:37.256 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:51:37.254618 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:51:37.269 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:51:37.268156 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:51:47.382 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:51:47.380344 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:51:47.394 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:51:47.393754 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:51:57.460 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:51:57.458674 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:51:57.472 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:51:57.470916 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:52:07.567 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:52:07.566286 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:52:07.578 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:52:07.576845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:52:17.689 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:52:17.688217 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:52:17.709 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:52:17.701676 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:52:27.805 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:52:27.803768 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:52:27.814 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:52:27.813122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:52:37.894 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:52:37.892545 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:52:37.907 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:52:37.906379 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:52:48.012 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:52:48.010701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:52:48.025 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:52:48.024661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:52:58.133 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:52:58.131560 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:52:58.144 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:52:58.143124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:53:08.259 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:53:08.258537 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:53:08.275 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:53:08.273903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:53:18.377 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:53:18.375683 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:53:18.394 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:53:18.393531 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:53:28.468 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:53:28.466655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:53:28.485 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:53:28.478960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:53:38.545 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:53:38.544235 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:53:38.600 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:53:38.599219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:53:48.750 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:53:48.749364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:53:48.761 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:53:48.759958 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:53:58.849 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:53:58.846833 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:53:58.861 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:53:58.860364 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:54:08.963 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:54:08.962000 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:54:08.976 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:54:08.975446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:54:19.052 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:54:19.050535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:54:19.065 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:54:19.064433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:54:29.179 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:54:29.178388 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:54:29.194 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:54:29.193279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:54:39.303 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:54:39.301471 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:54:39.315 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:54:39.314104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:54:49.393 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:54:49.392472 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:54:49.405 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:54:49.404408 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:54:59.520 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:54:59.519327 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:54:59.533 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:54:59.532795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:55:09.603 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:55:09.602195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:55:09.616 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:55:09.615675 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:55:19.748 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:55:19.747254 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:55:19.762 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:55:19.761309 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:55:29.859 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:55:29.858105 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:55:29.870 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:55:29.869204 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:55:40.025 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:55:40.023891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:55:40.040 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:55:40.038768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:55:50.111 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:55:50.109813 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:55:50.124 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:55:50.122857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:56:00.197 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:56:00.195905 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:56:00.210 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:56:00.208918 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:56:10.356 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:56:10.354571 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:56:10.367 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:56:10.366416 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:56:20.473 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:56:20.472190 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:56:20.488 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:56:20.487478 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:56:30.593 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:56:30.592066 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:56:30.643 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:56:30.641938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:56:40.713 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:56:40.711709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:56:40.724 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:56:40.723330 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:56:50.796 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:56:50.795408 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:56:50.811 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:56:50.809819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:57:00.940 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:57:00.939011 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:57:00.955 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:57:00.953932 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:57:11.059 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:57:11.058393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:57:11.073 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:57:11.072135 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:57:21.183 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:57:21.181862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:57:21.193 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:57:21.192419 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:57:31.283 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:57:31.281368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:57:31.294 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:57:31.293104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:57:41.363 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:57:41.361637 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:57:41.375 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:57:41.374735 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:57:51.437 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:57:51.433796 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:57:51.448 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:57:51.447328 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:58:01.521 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:58:01.519963 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:58:01.535 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:58:01.534091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:58:11.611 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:58:11.610551 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:58:11.624 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:58:11.622886 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:58:21.729 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:58:21.728022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:58:21.743 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:58:21.742313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:58:31.876 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:58:31.874942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:58:32.739 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:58:31.888508 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:58:41.987 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:58:41.986178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:58:42.000 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:58:41.999437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:58:52.152 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:58:52.150459 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:58:52.166 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:58:52.165005 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:59:02.285 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:59:02.284161 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:59:02.298 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:59:02.297139 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:59:12.407 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:59:12.406118 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:59:12.419 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:59:12.418291 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:59:22.508 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:59:22.506987 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:59:22.522 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:59:22.521091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:59:32.649 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:59:32.647689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:59:32.662 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:59:32.661541 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:59:42.765 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:59:42.763790 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:59:42.783 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:59:42.782003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:59:52.910 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:59:52.908533 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T16:59:52.921 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 16:59:52.920149 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:00:02.999 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:00:02.998401 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:00:03.011 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:00:03.009903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:00:13.077 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:00:13.076256 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:00:13.142 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:00:13.141183 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:00:23.269 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:00:23.268144 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:00:23.280 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:00:23.278974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:00:33.389 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:00:33.387402 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:00:33.402 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:00:33.400792 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:00:43.476 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:00:43.474765 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:00:43.490 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:00:43.489257 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:00:53.552 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:00:53.551136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:00:53.564 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:00:53.563592 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:01:03.663 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:01:03.662290 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:01:03.677 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:01:03.675916 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:01:13.802 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:01:13.801121 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:01:13.815 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:01:13.813746 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:01:23.925 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:01:23.923959 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:01:23.937 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:01:23.936196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:01:34.009 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:01:34.007592 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:01:34.025 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:01:34.024164 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:01:44.091 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:01:44.090312 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:01:44.106 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:01:44.105175 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:01:54.184 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:01:54.182998 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:01:54.199 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:01:54.198159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:02:04.270 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:02:04.269232 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:02:04.325 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:02:04.323883 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:02:14.387 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:02:14.386253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:02:14.442 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:02:14.440843 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:02:24.541 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:02:24.539442 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:02:24.553 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:02:24.552352 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:02:34.629 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:02:34.627408 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:02:34.641 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:02:34.640116 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:02:44.746 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:02:44.744378 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:02:44.759 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:02:44.757615 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:02:54.876 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:02:54.874309 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:02:54.890 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:02:54.889136 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:03:04.957 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:03:04.956219 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:03:04.970 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:03:04.968847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:03:15.039 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:03:15.038404 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:03:15.051 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:03:15.050673 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:03:25.173 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:03:25.171578 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:03:25.187 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:03:25.186466 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:03:35.255 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:03:35.253631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:03:35.267 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:03:35.266514 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:03:45.345 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:03:45.344335 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:03:45.435 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:03:45.434388 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:03:55.519 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:03:55.518300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:03:55.573 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:03:55.572390 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:04:05.668 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:04:05.666500 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:04:05.686 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:04:05.685054 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:04:15.765 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:04:15.763906 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:04:15.777 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:04:15.776609 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:04:25.846 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:04:25.844864 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:04:25.858 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:04:25.857718 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:04:27.838 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:04:27.837056 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T17:04:28.512 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:04:28.510030 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T17:04:28.497+0000 7f6db5c77380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T17:04:28.512 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T17:04:28.497+0000 7f6db5c77380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T17:04:28.512 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T17:04:28.497+0000 7f6db5c77380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T17:04:28.512 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T17:04:35.964 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:04:35.962942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:04:35.977 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:04:35.975787 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:04:46.050 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:04:46.048777 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:04:46.065 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:04:46.063751 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:04:56.165 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:04:56.164056 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:04:56.179 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:04:56.178545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:05:06.299 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:05:06.297731 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:05:06.312 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:05:06.311020 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:05:16.409 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:05:16.407850 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:05:16.459 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:05:16.458273 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:05:26.586 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:05:26.584310 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:05:26.602 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:05:26.600848 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:05:36.676 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:05:36.674700 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:05:36.689 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:05:36.687706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:05:46.793 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:05:46.791748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:05:46.805 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:05:46.803612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:05:56.882 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:05:56.881041 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:05:56.896 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:05:56.895022 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:06:07.011 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:06:07.009457 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:06:07.024 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:06:07.023462 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:06:17.086 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:06:17.084432 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:06:17.097 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:06:17.096429 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:06:27.170 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:06:27.168993 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:06:27.183 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:06:27.182639 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:06:37.279 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:06:37.278101 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:06:37.295 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:06:37.294350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:06:47.372 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:06:47.371139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:06:47.385 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:06:47.383824 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:06:57.470 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:06:57.469097 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:06:57.485 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:06:57.483594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:07:07.574 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:07:07.572422 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:07:07.585 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:07:07.583939 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:07:17.646 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:07:17.645109 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:07:17.657 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:07:17.656500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:07:27.725 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:07:27.723891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:07:27.738 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:07:27.736760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:07:37.906 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:07:37.904249 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:07:37.920 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:07:37.919652 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:07:48.014 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:07:48.012629 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:07:48.028 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:07:48.027294 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:07:58.096 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:07:58.095017 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:07:58.185 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:07:58.176671 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:08:08.277 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:08:08.276061 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:08:08.293 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:08:08.292241 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:08:18.363 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:08:18.361415 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:08:18.403 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:08:18.402019 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:08:28.477 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:08:28.476075 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:08:28.491 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:08:28.489419 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:08:38.594 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:08:38.593294 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:08:38.607 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:08:38.606195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:08:48.713 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:08:48.707245 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:08:48.719 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:08:48.718215 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:08:58.818 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:08:58.816751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:08:58.829 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:08:58.828396 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:09:08.918 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:09:08.916486 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:09:08.932 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:09:08.930943 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:09:19.060 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:09:19.058770 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:09:19.073 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:09:19.071755 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:09:29.136 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:09:29.134636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:09:29.203 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:09:29.201775 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:09:39.282 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:09:39.280206 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:09:39.333 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:09:39.332039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:09:49.513 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:09:49.511399 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:09:49.528 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:09:49.527441 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:09:59.665 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:09:59.663913 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:09:59.680 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:09:59.679478 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:10:09.827 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:10:09.826049 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:10:09.841 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:10:09.840348 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:10:19.923 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:10:19.922534 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:10:19.972 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:10:19.970407 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:10:30.068 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:10:30.067091 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:10:30.083 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:10:30.082199 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:10:40.329 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:10:40.328141 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:10:40.351 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:10:40.350221 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:10:50.427 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:10:50.425514 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:10:50.440 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:10:50.438929 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:11:00.534 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:11:00.533007 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:11:00.549 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:11:00.548068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:11:10.658 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:11:10.656269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:11:10.668 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:11:10.667239 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:11:20.787 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:11:20.785568 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:11:20.800 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:11:20.798974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:11:30.911 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:11:30.909917 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:11:30.924 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:11:30.923235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:11:41.009 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:11:41.007468 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:11:41.019 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:11:41.018142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:11:51.089 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:11:51.087704 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:11:51.101 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:11:51.100487 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:12:01.172 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:12:01.170469 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:12:01.228 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:12:01.227016 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:12:11.339 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:12:11.337676 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:12:11.352 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:12:11.350933 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:12:21.464 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:12:21.463134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:12:21.478 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:12:21.476782 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:12:31.604 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:12:31.602594 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:12:31.619 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:12:31.618682 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:12:41.741 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:12:41.740059 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:12:41.752 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:12:41.751569 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:12:51.877 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:12:51.875522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:12:51.914 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:12:51.911974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:13:02.017 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:13:02.015159 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:13:02.029 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:13:02.028394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:13:12.101 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:13:12.098747 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:13:12.113 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:13:12.112454 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:13:22.215 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:13:22.213551 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:13:22.234 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:13:22.232235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:13:32.351 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:13:32.349300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:13:32.363 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:13:32.361664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:13:42.494 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:13:42.492363 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:13:42.506 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:13:42.505266 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:13:52.585 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:13:52.583932 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:13:52.597 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:13:52.595835 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:14:02.711 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:14:02.709735 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:14:02.722 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:14:02.721425 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:14:12.842 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:14:12.841087 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:14:12.855 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:14:12.853933 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:14:22.972 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:14:22.970481 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:14:22.984 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:14:22.983153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:14:33.090 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:14:33.089092 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:14:33.101 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:14:33.100469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:14:43.185 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:14:43.183364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:14:43.239 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:14:43.238266 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:14:53.378 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:14:53.376969 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:14:53.397 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:14:53.395630 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:15:03.516 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:15:03.514516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:15:03.530 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:15:03.528989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:15:13.643 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:15:13.642147 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:15:13.658 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:15:13.656940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:15:23.779 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:15:23.778118 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:15:23.791 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:15:23.790093 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:15:33.875 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:15:33.873703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:15:33.890 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:15:33.889293 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:15:43.989 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:15:43.987628 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:15:44.000 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:15:43.999384 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:15:54.069 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:15:54.067227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:15:54.083 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:15:54.081936 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:16:04.162 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:16:04.160942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:16:04.218 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:16:04.217205 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:16:14.295 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:16:14.293395 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:16:14.306 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:16:14.305458 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:16:24.425 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:16:24.423043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:16:24.438 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:16:24.436710 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:16:34.508 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:16:34.506527 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:16:34.521 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:16:34.520438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:16:44.631 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:16:44.629246 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:16:44.644 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:16:44.643097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:16:54.724 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:16:54.723071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:16:54.736 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:16:54.735292 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:17:04.842 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:17:04.841061 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:17:04.856 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:17:04.854474 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:17:14.955 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:17:14.954091 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:17:14.967 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:17:14.965858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:17:25.075 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:17:25.073973 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:17:25.089 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:17:25.088120 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:17:35.156 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:17:35.154714 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:17:35.170 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:17:35.168746 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:17:45.243 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:17:45.241622 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:17:45.256 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:17:45.254835 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:17:55.379 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:17:55.377822 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:17:55.393 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:17:55.391607 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:18:05.507 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:18:05.505551 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:18:05.519 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:18:05.517948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:18:15.597 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:18:15.596253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:18:15.611 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:18:15.610277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:18:25.704 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:18:25.702696 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:18:25.717 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:18:25.715733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:18:35.840 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:18:35.838476 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:18:35.853 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:18:35.852144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:18:45.965 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:18:45.964277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:18:46.021 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:18:46.020098 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:18:56.210 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:18:56.209208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:18:56.223 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:18:56.222469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:19:06.334 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:19:06.332917 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:19:06.348 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:19:06.346512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:19:16.427 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:19:16.425248 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:19:16.438 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:19:16.437480 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:19:26.534 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:19:26.533222 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:19:26.546 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:19:26.545331 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:19:36.655 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:19:36.654004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:19:36.666 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:19:36.665288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:19:46.747 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:19:46.745692 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:19:46.760 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:19:46.759070 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:19:56.825 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:19:56.823452 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:19:56.836 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:19:56.834777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:20:06.938 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:20:06.936315 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:20:06.950 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:20:06.948735 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:20:17.020 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:20:17.018248 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:20:17.076 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:20:17.074525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:20:27.148 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:20:27.146833 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:20:27.161 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:20:27.160077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:20:37.266 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:20:37.264761 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:20:37.282 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:20:37.280994 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:20:47.349 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:20:47.347759 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:20:47.362 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:20:47.360683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:20:57.428 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:20:57.427186 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:20:57.444 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:20:57.443175 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:21:07.577 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:21:07.575403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:21:07.587 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:21:07.586129 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:21:08.522 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:21:08.520174 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T17:21:09.196 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:21:09.194487 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T17:21:09.181+0000 7fc5c992b380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T17:21:09.196 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T17:21:09.181+0000 7fc5c992b380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T17:21:09.197 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T17:21:09.181+0000 7fc5c992b380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T17:21:09.197 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T17:21:17.697 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:21:17.695378 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:21:17.711 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:21:17.710444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:21:27.781 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:21:27.779954 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:21:27.792 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:21:27.791406 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:21:37.887 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:21:37.885448 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:21:37.900 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:21:37.899243 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:21:47.967 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:21:47.966007 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:21:47.978 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:21:47.977402 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:21:58.053 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:21:58.051900 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:21:58.065 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:21:58.064366 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:22:08.148 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:22:08.146297 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:22:08.198 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:22:08.196390 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:22:18.298 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:22:18.296564 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:22:18.324 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:22:18.323042 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:22:28.445 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:22:28.444226 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:22:28.456 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:22:28.455543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:22:38.552 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:22:38.549653 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:22:38.564 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:22:38.562891 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:22:48.622 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:22:48.621167 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:22:48.633 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:22:48.632126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:22:58.710 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:22:58.709060 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:22:58.718 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:22:58.717496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:23:08.825 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:23:08.823694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:23:08.840 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:23:08.838977 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:23:18.959 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:23:18.957226 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:23:18.968 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:23:18.966917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:23:29.103 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:23:29.101954 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:23:29.116 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:23:29.115308 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:23:39.183 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:23:39.182129 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:23:39.196 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:23:39.194683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:23:49.315 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:23:49.313230 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:23:49.327 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:23:49.326303 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:23:59.402 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:23:59.401091 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:23:59.456 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:23:59.454942 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:24:09.538 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:24:09.536603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:24:09.582 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:24:09.581631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:24:19.699 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:24:19.697439 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:24:19.711 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:24:19.710539 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:24:29.809 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:24:29.807711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:24:29.821 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:24:29.820447 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:24:39.933 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:24:39.931489 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:24:39.947 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:24:39.945875 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:24:50.088 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:24:50.086534 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:24:50.099 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:24:50.098541 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:25:00.175 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:25:00.174015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:25:00.190 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:25:00.188702 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:25:10.291 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:25:10.289918 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:25:10.304 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:25:10.303309 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:25:20.414 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:25:20.412747 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:25:20.428 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:25:20.426963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:25:30.523 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:25:30.521682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:25:30.536 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:25:30.535002 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:25:40.647 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:25:40.646314 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:25:40.660 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:25:40.658973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:25:50.750 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:25:50.748868 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:25:50.763 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:25:50.762306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:26:00.841 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:26:00.840129 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:26:00.852 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:26:00.849596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:26:10.929 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:26:10.927646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:26:10.942 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:26:10.941015 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:26:21.079 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:26:21.078096 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:26:21.095 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:26:21.093660 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:26:31.157 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:26:31.155697 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:26:31.171 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:26:31.170207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:26:41.250 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:26:41.248718 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:26:41.264 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:26:41.261403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:26:51.332 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:26:51.330996 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:26:51.344 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:26:51.343289 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:27:01.455 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:27:01.453121 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:27:01.467 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:27:01.465970 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:27:11.580 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:27:11.578396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:27:11.593 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:27:11.591989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:27:21.714 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:27:21.712496 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:27:21.726 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:27:21.725436 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:27:31.825 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:27:31.823874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:27:31.843 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:27:31.841811 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:27:41.970 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:27:41.968666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:27:41.981 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:27:41.980466 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:27:52.069 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:27:52.068001 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:27:52.083 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:27:52.081822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:28:02.174 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:28:02.173060 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:28:02.226 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:28:02.224507 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:28:12.306 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:28:12.304256 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:28:12.318 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:28:12.316650 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:28:22.400 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:28:22.399285 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:28:22.470 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:28:22.468580 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:28:32.548 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:28:32.545938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:28:32.562 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:28:32.560471 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:28:42.671 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:28:42.670054 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:28:42.684 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:28:42.682681 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:28:52.805 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:28:52.803756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:28:52.818 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:28:52.817286 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:29:02.956 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:29:02.954838 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:29:02.971 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:29:02.970323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:29:13.036 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:29:13.034494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:29:13.221 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:29:13.046709 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:29:23.152 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:29:23.150574 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:29:23.165 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:29:23.164081 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:29:33.266 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:29:33.265035 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:29:33.278 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:29:33.276627 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:29:43.390 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:29:43.388494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:29:43.402 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:29:43.400669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:29:53.534 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:29:53.532506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:29:53.554 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:29:53.553394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:30:03.712 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:30:03.710518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:30:03.724 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:30:03.722646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:30:13.827 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:30:13.825729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:30:13.840 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:30:13.838808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:30:23.974 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:30:23.972995 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:30:23.988 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:30:23.987240 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:30:34.091 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:30:34.090270 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:30:34.104 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:30:34.102625 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:30:44.182 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:30:44.180871 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:30:44.198 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:30:44.196749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:30:54.312 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:30:54.310308 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:30:54.326 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:30:54.324652 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:31:04.452 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:31:04.450760 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:31:04.467 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:31:04.465530 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:31:14.533 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:31:14.531300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:31:14.545 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:31:14.544274 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:31:24.649 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:31:24.647958 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:31:24.662 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:31:24.660486 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:31:34.733 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:31:34.731302 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:31:34.776 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:31:34.775057 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:31:44.856 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:31:44.855163 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:31:44.868 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:31:44.867389 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:31:55.027 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:31:55.025399 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:31:55.035 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:31:55.034138 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:32:05.168 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:32:05.166235 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:32:05.182 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:32:05.180690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:32:15.285 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:32:15.284022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:32:15.301 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:32:15.299571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:32:25.397 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:32:25.395964 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:32:25.412 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:32:25.410143 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:32:35.529 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:32:35.527347 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:32:35.543 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:32:35.542356 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:32:45.648 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:32:45.646915 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:32:45.662 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:32:45.660887 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:32:55.738 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:32:55.737059 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:32:55.753 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:32:55.751670 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:33:05.848 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:33:05.846937 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:33:05.861 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:33:05.859505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:33:15.947 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:33:15.945079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:33:15.960 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:33:15.959492 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:33:26.029 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:33:26.027950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:33:26.044 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:33:26.042748 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:33:36.120 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:33:36.118794 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:33:36.134 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:33:36.132839 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:33:46.250 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:33:46.248039 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:33:46.264 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:33:46.263227 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:33:56.373 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:33:56.371339 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:33:56.385 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:33:56.383791 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:34:06.496 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:34:06.494614 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:34:06.506 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:34:06.504846 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:34:16.609 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:34:16.608311 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:34:16.620 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:34:16.619077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:34:26.740 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:34:26.733390 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:34:26.746 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:34:26.744682 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:34:36.874 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:34:36.872854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:34:36.888 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:34:36.886864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:34:47.012 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:34:47.011095 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:34:47.022 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:34:47.020563 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:34:57.095 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:34:57.093481 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:34:57.108 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:34:57.106765 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:35:07.212 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:35:07.211101 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:35:07.225 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:35:07.223775 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:35:17.310 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:35:17.308977 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:35:17.323 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:35:17.321683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:35:27.400 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:35:27.399071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:35:27.416 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:35:27.414923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:35:37.485 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:35:37.484147 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:35:37.497 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:35:37.495794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:35:47.554 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:35:47.552760 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:35:47.568 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:35:47.567083 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:35:57.671 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:35:57.670067 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:35:57.689 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:35:57.687771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:36:07.790 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:36:07.788168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:36:07.801 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:36:07.800184 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:36:17.886 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:36:17.884866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:36:17.901 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:36:17.899606 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:36:28.033 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:36:28.031494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:36:28.047 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:36:28.046286 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:36:38.196 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:36:38.194691 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:36:38.211 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:36:38.210191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:36:48.317 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:36:48.315381 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:36:48.333 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:36:48.331577 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:36:58.416 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:36:58.415082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:36:58.470 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:36:58.469022 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:37:08.543 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:37:08.541445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:37:08.554 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:37:08.553027 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:37:18.695 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:37:18.693832 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:37:18.712 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:37:18.710412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:37:28.788 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:37:28.787132 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:37:28.803 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:37:28.802307 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:37:38.883 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:37:38.881633 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:37:38.896 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:37:38.894489 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:37:48.974 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:37:48.972484 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:37:48.990 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:37:48.988640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:37:49.203 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:37:49.201891 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T17:37:49.941 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:37:49.939648 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T17:37:49.928+0000 7fb678608380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T17:37:49.942 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T17:37:49.928+0000 7fb678608380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T17:37:49.942 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T17:37:49.928+0000 7fb678608380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T17:37:49.942 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T17:37:59.060 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:37:59.058522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:37:59.073 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:37:59.072484 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:38:09.152 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:38:09.150226 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:38:09.165 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:38:09.163945 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:38:19.237 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:38:19.236177 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:38:19.252 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:38:19.251314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:38:29.372 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:38:29.370684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:38:29.386 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:38:29.384626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:38:39.481 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:38:39.479882 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:38:39.495 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:38:39.494580 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:38:49.636 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:38:49.634288 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:38:49.648 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:38:49.647092 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:38:59.751 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:38:59.749815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:38:59.762 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:38:59.761190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:39:09.833 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:39:09.831596 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:39:09.845 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:39:09.844454 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:39:19.951 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:39:19.949239 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:39:19.964 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:39:19.963175 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:39:30.075 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:39:30.073160 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:39:30.087 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:39:30.086450 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:39:40.260 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:39:40.258890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:39:40.273 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:39:40.271650 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:39:50.482 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:39:50.480380 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:39:50.492 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:39:50.490883 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:40:00.602 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:40:00.601046 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:40:00.613 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:40:00.611532 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:40:10.727 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:40:10.725132 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:40:10.740 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:40:10.738775 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:40:20.818 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:40:20.816119 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:40:20.831 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:40:20.829816 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:40:30.902 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:40:30.900264 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:40:30.960 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:40:30.958302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:40:41.038 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:40:41.036439 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:40:41.051 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:40:41.050138 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:40:51.118 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:40:51.116091 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:40:51.131 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:40:51.129899 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:41:01.243 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:41:01.241873 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:41:01.256 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:41:01.254547 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:41:11.348 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:41:11.346514 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:41:11.362 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:41:11.360813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:41:21.444 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:41:21.438064 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:41:21.450 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:41:21.449213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:41:31.552 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:41:31.550316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:41:31.562 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:41:31.561172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:41:41.617 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:41:41.615926 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:41:41.626 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:41:41.625049 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:41:51.720 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:41:51.718449 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:41:51.729 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:41:51.728001 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:42:01.821 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:42:01.819439 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:42:01.832 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:42:01.831284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:42:11.899 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:42:11.897637 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:42:11.912 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:42:11.910851 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:42:22.027 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:42:22.026100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:42:22.042 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:42:22.041098 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:42:32.112 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:42:32.110893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:42:32.125 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:42:32.123964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:42:42.190 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:42:42.188606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:42:42.204 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:42:42.202850 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:42:52.271 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:42:52.269804 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:42:52.285 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:42:52.284011 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:43:02.371 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:43:02.369357 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:43:02.383 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:43:02.382216 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:43:12.497 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:43:12.495131 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:43:12.507 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:43:12.506085 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:43:22.640 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:43:22.638235 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:43:22.652 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:43:22.650962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:43:32.774 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:43:32.772979 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:43:32.787 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:43:32.786081 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:43:42.918 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:43:42.916095 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:43:42.929 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:43:42.927553 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:43:53.000 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:43:52.998498 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:43:53.013 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:43:53.012074 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:44:03.161 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:44:03.159509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:44:03.174 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:44:03.173139 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:44:13.278 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:44:13.276592 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:44:13.292 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:44:13.290806 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:44:23.391 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:44:23.389802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:44:23.406 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:44:23.404758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:44:33.517 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:44:33.515254 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:44:33.529 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:44:33.527669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:44:43.602 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:44:43.600584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:44:43.617 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:44:43.615912 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:44:53.705 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:44:53.703978 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:44:53.758 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:44:53.756584 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:45:03.882 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:45:03.880579 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:45:03.897 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:45:03.895172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:45:14.154 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:45:14.152642 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:45:14.167 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:45:14.166193 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:45:24.287 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:45:24.286037 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:45:24.300 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:45:24.299157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:45:34.387 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:45:34.386137 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:45:34.401 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:45:34.399714 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:45:44.511 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:45:44.509999 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:45:44.525 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:45:44.524092 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:45:54.641 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:45:54.639875 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:45:54.652 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:45:54.651095 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:46:04.775 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:46:04.773305 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:46:04.788 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:46:04.786813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:46:14.892 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:46:14.890780 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:46:14.903 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:46:14.902240 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:46:24.970 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:46:24.969075 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:46:24.983 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:46:24.981596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:46:35.056 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:46:35.054854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:46:35.075 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:46:35.073546 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:46:45.149 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:46:45.147800 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:46:45.207 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:46:45.205118 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:46:55.327 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:46:55.325610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:46:55.339 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:46:55.338238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:47:05.415 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:47:05.413676 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:47:05.461 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:47:05.459784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:47:15.575 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:47:15.573647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:47:15.588 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:47:15.586517 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:47:25.704 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:47:25.702933 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:47:25.718 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:47:25.716527 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:47:35.809 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:47:35.807044 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:47:35.824 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:47:35.822712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:47:45.895 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:47:45.892929 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:47:46.237 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:47:45.907455 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:47:55.996 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:47:55.994971 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:47:56.010 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:47:56.009105 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:48:06.120 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:48:06.118522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:48:06.142 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:48:06.141011 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:48:16.245 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:48:16.243702 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:48:16.258 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:48:16.256684 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:48:26.345 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:48:26.341984 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:48:26.366 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:48:26.365187 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:48:36.473 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:48:36.471259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:48:36.486 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:48:36.485019 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:48:46.592 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:48:46.590132 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:48:46.605 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:48:46.604344 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:48:56.711 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:48:56.709862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:48:56.725 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:48:56.723988 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:49:06.795 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:49:06.793948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:49:06.809 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:49:06.807688 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:49:16.890 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:49:16.888204 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:49:16.900 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:49:16.899370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:49:27.040 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:49:27.038659 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:49:27.056 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:49:27.054285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:49:37.129 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:49:37.127433 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:49:37.143 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:49:37.141666 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:49:47.244 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:49:47.242546 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:49:47.254 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:49:47.252845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:49:57.339 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:49:57.337261 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:49:57.351 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:49:57.349988 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:50:07.503 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:50:07.501459 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:50:07.517 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:50:07.515871 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:50:17.622 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:50:17.620817 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:50:17.637 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:50:17.635585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:50:27.699 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:50:27.697610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:50:27.712 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:50:27.711146 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:50:37.780 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:50:37.778414 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:50:37.795 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:50:37.794199 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:50:47.892 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:50:47.890613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:50:47.906 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:50:47.904605 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:50:58.010 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:50:58.008599 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:50:58.021 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:50:58.020224 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:51:08.129 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:51:08.127792 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:51:08.142 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:51:08.140583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:51:18.292 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:51:18.290280 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:51:18.313 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:51:18.311708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:51:28.420 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:51:28.418421 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:51:28.435 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:51:28.433502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:51:38.523 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:51:38.521443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:51:38.536 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:51:38.534788 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:51:48.650 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:51:48.648519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:51:48.663 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:51:48.662180 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:51:58.799 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:51:58.796996 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:51:58.814 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:51:58.812811 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:52:08.883 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:52:08.881311 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:52:08.897 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:52:08.895680 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:52:18.974 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:52:18.972509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:52:18.988 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:52:18.986499 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:52:29.097 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:52:29.095371 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:52:29.112 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:52:29.110579 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:52:39.183 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:52:39.181364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:52:39.196 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:52:39.195065 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:52:49.267 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:52:49.265447 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:52:49.310 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:52:49.309302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:52:59.392 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:52:59.390862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:52:59.404 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:52:59.403207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:53:09.526 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:53:09.524727 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:53:09.537 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:53:09.535496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:53:19.601 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:53:19.599914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:53:19.612 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:53:19.610676 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:53:29.683 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:53:29.681736 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:53:29.695 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:53:29.693473 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:53:39.798 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:53:39.796244 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:53:39.810 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:53:39.809160 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:53:49.896 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:53:49.894370 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:53:49.911 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:53:49.909632 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:53:59.981 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:53:59.979193 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:53:59.994 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:53:59.992832 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:54:10.066 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:54:10.064205 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:54:10.079 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:54:10.077962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:54:20.156 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:54:20.154349 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:54:20.168 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:54:20.167155 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:54:29.955 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:54:29.953386 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T17:54:30.232 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:54:30.230967 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:54:30.246 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:54:30.245091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:54:30.671 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:54:30.668851 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T17:54:30.655+0000 7f62d9abb380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T17:54:30.671 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T17:54:30.655+0000 7f62d9abb380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T17:54:30.671 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T17:54:30.655+0000 7f62d9abb380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T17:54:30.671 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T17:54:40.325 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:54:40.323256 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:54:40.336 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:54:40.334553 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:54:50.407 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:54:50.406036 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:54:50.420 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:54:50.418513 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:55:00.544 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:55:00.543072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:55:00.559 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:55:00.557578 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:55:10.631 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:55:10.629323 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:55:10.643 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:55:10.641693 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:55:20.746 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:55:20.744152 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:55:20.757 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:55:20.755576 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:55:30.825 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:55:30.823074 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:55:30.837 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:55:30.836039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:55:40.902 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:55:40.900392 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:55:40.912 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:55:40.911226 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:55:51.058 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:55:51.056289 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:55:51.073 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:55:51.072267 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:56:01.150 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:56:01.148934 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:56:01.215 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:56:01.214316 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:56:11.315 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:56:11.313800 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:56:11.329 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:56:11.327169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:56:21.402 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:56:21.400103 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:56:21.477 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:56:21.475196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:56:31.590 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:56:31.588231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:56:31.606 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:56:31.604720 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:56:41.698 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:56:41.696850 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:56:41.713 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:56:41.710343 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:56:51.846 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:56:51.844879 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:56:51.858 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:56:51.856948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:57:01.932 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:57:01.930970 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:57:01.945 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:57:01.943040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:57:12.059 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:57:12.058032 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:57:12.076 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:57:12.075049 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:57:22.143 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:57:22.142106 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:57:22.158 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:57:22.156584 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:57:32.257 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:57:32.255749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:57:32.270 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:57:32.268951 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:57:42.362 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:57:42.360407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:57:42.371 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:57:42.369972 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:57:52.442 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:57:52.440609 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:57:52.457 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:57:52.455423 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:58:02.557 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:58:02.555832 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:58:02.573 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:58:02.571214 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:58:12.647 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:58:12.645360 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:58:12.661 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:58:12.659326 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:58:22.811 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:58:22.809316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:58:22.823 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:58:22.821993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:58:33.006 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:58:33.004729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:58:33.022 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:58:33.020054 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:58:43.195 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:58:43.193635 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:58:43.211 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:58:43.210301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:58:53.293 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:58:53.291216 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:58:53.308 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:58:53.307398 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:59:03.427 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:59:03.425872 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:59:03.442 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:59:03.440403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:59:13.542 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:59:13.540238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:59:13.561 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:59:13.560218 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:59:23.677 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:59:23.675230 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:59:23.686 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:59:23.684758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:59:33.789 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:59:33.787307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:59:33.801 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:59:33.799959 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:59:43.863 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:59:43.861165 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:59:43.913 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:59:43.911367 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:59:54.031 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:59:54.029924 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T17:59:54.044 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 17:59:54.042557 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:00:04.149 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:00:04.147515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:00:04.158 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:00:04.156496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:00:14.265 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:00:14.264030 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:00:14.278 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:00:14.276619 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:00:24.390 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:00:24.387743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:00:24.402 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:00:24.401096 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:00:34.479 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:00:34.476948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:00:34.492 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:00:34.490125 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:00:44.569 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:00:44.567870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:00:44.587 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:00:44.580936 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:00:54.698 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:00:54.696363 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:00:54.714 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:00:54.712336 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:01:04.810 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:01:04.808287 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:01:04.824 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:01:04.822467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:01:14.906 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:01:14.904432 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:01:14.921 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:01:14.919500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:01:24.989 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:01:24.987663 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:01:24.998 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:01:24.996443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:01:35.109 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:01:35.107616 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:01:35.121 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:01:35.119933 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:01:45.192 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:01:45.190300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:01:45.240 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:01:45.238967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:01:55.310 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:01:55.308635 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:01:55.325 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:01:55.323212 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:02:05.431 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:02:05.429208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:02:05.445 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:02:05.442865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:02:15.522 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:02:15.520244 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:02:15.534 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:02:15.532590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:02:25.653 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:02:25.651187 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:02:25.670 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:02:25.668239 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:02:35.776 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:02:35.774681 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:02:35.789 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:02:35.787987 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:02:45.901 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:02:45.899712 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:02:45.914 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:02:45.912849 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:02:56.007 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:02:56.005914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:02:56.020 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:02:56.018832 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:03:06.084 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:03:06.082439 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:03:06.134 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:03:06.132299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:03:16.255 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:03:16.253771 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:03:16.267 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:03:16.265981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:03:26.338 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:03:26.336097 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:03:26.349 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:03:26.347563 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:03:36.458 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:03:36.456687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:03:36.472 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:03:36.471040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:03:46.540 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:03:46.538847 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:03:46.603 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:03:46.601524 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:03:56.697 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:03:56.695990 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:03:56.708 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:03:56.707254 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:04:06.789 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:04:06.787674 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:04:06.801 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:04:06.799548 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:04:16.947 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:04:16.944711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:04:16.961 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:04:16.959531 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:04:27.081 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:04:27.080058 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:04:27.091 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:04:27.089966 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:04:37.159 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:04:37.157970 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:04:37.172 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:04:37.171346 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:04:47.260 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:04:47.258922 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:04:47.276 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:04:47.274582 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:04:57.413 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:04:57.411430 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:04:57.425 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:04:57.423874 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:05:07.508 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:05:07.506131 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:05:07.523 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:05:07.521423 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:05:17.603 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:05:17.601609 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:05:17.618 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:05:17.616740 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:05:27.733 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:05:27.731492 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:05:27.742 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:05:27.741241 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:05:37.810 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:05:37.808935 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:05:37.825 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:05:37.823797 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:05:47.939 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:05:47.937046 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:05:47.953 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:05:47.951259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:05:58.032 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:05:58.030876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:05:58.047 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:05:58.046279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:06:08.156 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:06:08.153930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:06:08.169 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:06:08.167725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:06:18.289 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:06:18.286951 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:06:18.303 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:06:18.301230 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:06:28.377 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:06:28.375398 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:06:28.390 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:06:28.388687 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:06:38.511 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:06:38.509442 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:06:38.523 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:06:38.521749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:06:48.661 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:06:48.659030 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:06:48.675 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:06:48.673850 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:06:58.779 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:06:58.777916 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:06:58.792 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:06:58.790868 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:07:08.873 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:07:08.871505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:07:08.889 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:07:08.887291 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:07:18.969 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:07:18.967622 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:07:19.032 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:07:19.030978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:07:29.094 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:07:29.092189 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:07:29.106 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:07:29.105077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:07:39.173 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:07:39.171664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:07:39.188 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:07:39.186928 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:07:49.257 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:07:49.255496 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:07:49.306 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:07:49.303978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:07:59.391 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:07:59.389509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:07:59.440 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:07:59.438925 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:08:09.552 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:08:09.550479 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:08:09.569 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:08:09.568145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:08:19.664 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:08:19.662178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:08:19.682 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:08:19.680699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:08:29.795 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:08:29.793656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:08:29.807 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:08:29.806245 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:08:39.931 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:08:39.929825 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:08:39.947 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:08:39.946272 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:08:50.030 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:08:50.028490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:08:50.076 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:08:50.075323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:09:00.144 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:09:00.142864 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:09:00.156 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:09:00.154876 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:09:10.304 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:09:10.302107 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:09:10.316 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:09:10.315166 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:09:20.418 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:09:20.416179 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:09:20.432 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:09:20.430597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:09:30.508 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:09:30.506726 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:09:30.569 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:09:30.567124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:09:40.667 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:09:40.665827 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:09:40.682 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:09:40.680480 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:09:50.744 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:09:50.743067 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:09:50.760 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:09:50.758224 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:10:00.850 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:10:00.848880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:10:00.864 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:10:00.862679 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:10:10.987 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:10:10.985367 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:10:11.000 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:10:10.999166 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:10:21.074 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:10:21.072124 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:10:21.086 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:10:21.084662 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:10:31.209 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:10:31.206863 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:10:31.224 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:10:31.222758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:10:41.322 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:10:41.320545 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:10:41.336 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:10:41.334744 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:10:51.449 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:10:51.447864 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:10:51.461 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:10:51.460185 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:11:01.569 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:11:01.567646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:11:01.585 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:11:01.583121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:11:10.678 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:11:10.676172 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T18:11:11.388 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:11:11.386294 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T18:11:11.343+0000 7f6cb5dde380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T18:11:11.389 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T18:11:11.343+0000 7f6cb5dde380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T18:11:11.389 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T18:11:11.343+0000 7f6cb5dde380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T18:11:11.389 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T18:11:11.647 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:11:11.645560 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:11:11.658 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:11:11.657428 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:11:21.775 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:11:21.773162 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:11:21.785 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:11:21.784068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:11:31.851 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:11:31.848991 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:11:31.869 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:11:31.867879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:11:41.950 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:11:41.948583 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:11:41.963 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:11:41.962173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:11:52.079 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:11:52.077025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:11:52.093 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:11:52.091584 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:12:02.373 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:12:02.371753 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:12:02.384 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:12:02.382795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:12:12.467 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:12:12.465088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:12:12.514 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:12:12.513099 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:12:22.587 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:12:22.585105 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:12:22.602 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:12:22.600760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:12:32.687 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:12:32.684463 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:12:32.744 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:12:32.742728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:12:42.840 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:12:42.838321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:12:42.854 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:12:42.852932 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:12:52.925 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:12:52.923430 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:12:52.939 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:12:52.938300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:13:03.063 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:13:03.061830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:13:03.078 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:13:03.076812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:13:13.193 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:13:13.191018 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:13:13.207 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:13:13.205652 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:13:23.277 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:13:23.275146 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:13:23.288 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:13:23.286495 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:13:33.426 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:13:33.424484 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:13:33.440 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:13:33.439038 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:13:43.554 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:13:43.552210 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:13:43.567 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:13:43.565404 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:13:53.670 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:13:53.668604 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:13:53.683 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:13:53.681648 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:14:03.764 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:14:03.762705 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:14:03.781 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:14:03.779318 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:14:13.845 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:14:13.843335 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:14:13.858 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:14:13.856803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:14:23.932 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:14:23.930440 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:14:23.947 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:14:23.945571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:14:34.046 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:14:34.044575 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:14:34.060 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:14:34.059238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:14:44.133 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:14:44.131860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:14:44.147 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:14:44.145607 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:14:54.246 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:14:54.243882 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:14:54.260 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:14:54.258623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:15:04.334 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:15:04.332139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:15:04.348 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:15:04.346140 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:15:14.452 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:15:14.449838 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:15:14.467 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:15:14.464983 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:15:24.543 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:15:24.541556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:15:24.556 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:15:24.555139 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:15:34.672 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:15:34.670179 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:15:34.686 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:15:34.685048 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:15:44.801 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:15:44.799196 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:15:44.812 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:15:44.811090 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:15:54.892 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:15:54.890892 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:15:54.956 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:15:54.954249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:16:05.061 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:16:05.059717 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:16:05.072 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:16:05.071091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:16:15.189 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:16:15.186807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:16:15.202 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:16:15.200417 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:16:25.303 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:16:25.301360 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:16:25.315 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:16:25.313322 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:16:35.386 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:16:35.384480 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:16:35.436 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:16:35.434603 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:16:45.542 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:16:45.540718 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:16:45.555 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:16:45.553374 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:16:55.624 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:16:55.622330 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:16:55.635 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:16:55.634138 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:17:05.721 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:17:05.719242 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:17:05.735 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:17:05.733537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:17:15.813 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:17:15.811103 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:17:15.876 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:17:15.874255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:17:25.954 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:17:25.952451 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:17:25.968 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:17:25.966825 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:17:36.113 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:17:36.111594 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:17:36.123 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:17:36.121271 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:17:46.238 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:17:46.236006 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:17:46.253 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:17:46.252024 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:17:56.407 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:17:56.404942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:17:56.417 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:17:56.415690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:18:06.624 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:18:06.622162 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:18:06.692 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:18:06.690881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:18:16.778 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:18:16.746649 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:18:16.813 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:18:16.812253 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:18:26.962 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:18:26.960613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:18:26.973 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:18:26.972244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:18:37.052 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:18:37.050196 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:18:37.063 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:18:37.061537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:18:47.171 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:18:47.169494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:18:47.204 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:18:47.202184 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:18:57.323 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:18:57.321285 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:18:57.337 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:18:57.335597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:19:07.404 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:19:07.401850 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:19:07.417 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:19:07.415220 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:19:17.548 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:19:17.546913 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:19:17.563 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:19:17.561784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:19:27.647 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:19:27.645259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:19:27.657 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:19:27.656364 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:19:37.757 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:19:37.755357 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:19:37.768 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:19:37.767118 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:19:47.870 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:19:47.868127 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:19:47.886 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:19:47.884414 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:19:57.992 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:19:57.990213 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:19:58.004 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:19:58.002815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:20:08.077 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:20:08.075140 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:20:08.089 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:20:08.087237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:20:18.198 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:20:18.196055 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:20:18.212 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:20:18.210382 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:20:28.325 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:20:28.323854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:20:28.338 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:20:28.336991 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:20:38.411 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:20:38.409689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:20:38.424 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:20:38.422404 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:20:48.495 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:20:48.493584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:20:48.506 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:20:48.504566 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:20:58.629 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:20:58.626888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:20:58.641 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:20:58.639487 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:21:08.705 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:21:08.703240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:21:08.720 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:21:08.718673 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:21:18.800 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:21:18.798331 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:21:18.820 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:21:18.818417 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:21:28.892 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:21:28.890118 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:21:28.983 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:21:28.981914 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:21:39.135 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:21:39.133238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:21:39.147 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:21:39.145033 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:21:49.217 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:21:49.215619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:21:49.228 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:21:49.227193 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:21:59.298 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:21:59.297002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:21:59.351 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:21:59.349438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:22:09.456 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:22:09.454498 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:22:09.469 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:22:09.468123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:22:19.583 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:22:19.581428 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:22:19.596 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:22:19.594850 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:22:29.667 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:22:29.665793 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:22:29.680 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:22:29.678870 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:22:39.768 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:22:39.766221 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:22:39.780 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:22:39.778661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:22:49.853 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:22:49.851682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:22:49.865 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:22:49.863197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:22:59.943 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:22:59.940939 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:23:00.009 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:23:00.007714 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:23:10.135 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:23:10.133661 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:23:10.148 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:23:10.146412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:23:20.280 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:23:20.277826 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:23:20.292 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:23:20.290559 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:23:30.388 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:23:30.386392 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:23:30.400 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:23:30.398815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:23:40.506 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:23:40.504437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:23:40.519 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:23:40.517745 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:23:50.653 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:23:50.650999 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:23:50.666 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:23:50.664896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:24:00.784 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:24:00.782388 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:24:00.799 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:24:00.797183 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:24:10.876 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:24:10.874114 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:24:10.887 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:24:10.885873 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:24:20.952 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:24:20.950310 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:24:20.996 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:24:20.994213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:24:31.126 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:24:31.124668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:24:31.138 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:24:31.136504 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:24:41.252 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:24:41.250608 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:24:41.267 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:24:41.265976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:24:51.382 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:24:51.380729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:24:51.396 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:24:51.394371 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:25:01.489 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:25:01.486965 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:25:01.501 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:25:01.499728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:25:11.579 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:25:11.576955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:25:11.595 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:25:11.593720 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:25:21.701 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:25:21.699041 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:25:21.714 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:25:21.712512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:25:31.779 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:25:31.777792 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:25:31.788 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:25:31.787011 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:25:41.898 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:25:41.896182 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:25:41.910 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:25:41.909223 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:25:51.985 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:25:51.984002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:25:52.048 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:25:52.046755 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:26:02.173 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:26:02.171136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:26:02.367 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:26:02.184026 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:26:12.299 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:26:12.296949 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:26:12.311 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:26:12.309157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:26:22.413 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:26:22.411186 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:26:22.426 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:26:22.424268 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:26:32.552 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:26:32.549687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:26:32.634 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:26:32.562919 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:26:42.664 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:26:42.662206 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:26:42.677 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:26:42.675243 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:26:52.782 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:26:52.779920 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:26:52.794 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:26:52.793268 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:27:02.874 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:27:02.871914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:27:02.922 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:27:02.920449 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:27:12.999 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:27:12.997433 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:27:13.012 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:27:13.011114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:27:23.167 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:27:23.165751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:27:23.182 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:27:23.180418 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:27:33.349 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:27:33.346710 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:27:33.360 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:27:33.359314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:27:43.423 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:27:43.421220 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:27:43.437 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:27:43.435157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:27:51.398 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:27:51.396087 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T18:27:52.092 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:27:52.090297 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T18:27:52.078+0000 7f25eb053380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T18:27:52.093 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T18:27:52.078+0000 7f25eb053380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T18:27:52.093 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T18:27:52.078+0000 7f25eb053380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T18:27:52.093 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T18:27:53.546 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:27:53.544706 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:27:53.560 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:27:53.558678 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:28:03.651 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:28:03.649730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:28:03.663 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:28:03.661650 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:28:13.732 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:28:13.727157 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:28:13.742 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:28:13.740592 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:28:23.881 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:28:23.879943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:28:23.896 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:28:23.894297 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:28:33.997 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:28:33.995540 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:28:34.011 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:28:34.009665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:28:44.110 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:28:44.108062 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:28:44.141 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:28:44.139727 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:28:54.255 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:28:54.253528 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:28:54.268 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:28:54.267172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:29:04.337 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:29:04.335062 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:29:04.350 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:29:04.348715 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:29:14.481 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:29:14.479077 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:29:14.498 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:29:14.496989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:29:24.569 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:29:24.567273 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:29:24.583 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:29:24.581740 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:29:34.664 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:29:34.662155 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:29:34.725 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:29:34.717764 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:29:44.820 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:29:44.818276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:29:44.856 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:29:44.854783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:29:54.926 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:29:54.924453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:29:54.938 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:29:54.937114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:30:05.007 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:30:05.005759 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:30:05.054 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:30:05.052646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:30:15.115 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:30:15.113683 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:30:15.128 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:30:15.126846 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:30:25.233 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:30:25.231103 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:30:25.246 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:30:25.244587 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:30:35.356 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:30:35.354328 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:30:35.370 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:30:35.368702 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:30:45.444 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:30:45.442578 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:30:45.455 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:30:45.453625 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:30:55.590 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:30:55.587788 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:30:55.612 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:30:55.610808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:31:05.679 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:31:05.677228 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:31:05.753 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:31:05.751355 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:31:15.903 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:31:15.901659 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:31:15.917 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:31:15.915994 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:31:25.989 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:31:25.987930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:31:26.002 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:31:26.000430 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:31:36.085 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:31:36.083206 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:31:36.099 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:31:36.098147 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:31:46.197 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:31:46.195136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:31:46.211 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:31:46.209121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:31:56.310 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:31:56.308437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:31:56.323 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:31:56.322284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:32:06.439 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:32:06.436962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:32:06.456 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:32:06.454389 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:32:16.566 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:32:16.564539 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:32:16.576 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:32:16.574904 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:32:26.694 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:32:26.692162 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:32:26.706 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:32:26.704704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:32:36.813 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:32:36.811753 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:32:36.827 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:32:36.825429 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:32:46.937 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:32:46.934809 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:32:46.949 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:32:46.948307 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:32:57.044 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:32:57.042049 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:32:57.058 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:32:57.056727 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:33:07.132 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:33:07.130821 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:33:07.146 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:33:07.145163 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:33:17.221 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:33:17.218972 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:33:17.291 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:33:17.289274 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:33:27.361 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:33:27.359229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:33:27.375 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:33:27.373560 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:33:37.493 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:33:37.491369 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:33:37.587 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:33:37.502704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:33:47.574 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:33:47.572208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:33:47.583 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:33:47.581984 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:33:57.698 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:33:57.696048 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:33:57.710 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:33:57.708684 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:34:07.819 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:34:07.817270 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:34:07.833 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:34:07.831537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:34:17.947 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:34:17.945323 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:34:17.960 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:34:17.958813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:34:28.039 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:34:28.037063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:34:28.052 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:34:28.050722 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:34:38.161 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:34:38.159834 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:34:38.193 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:34:38.191212 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:34:48.309 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:34:48.307562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:34:48.321 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:34:48.320059 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:34:58.400 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:34:58.398526 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:34:58.415 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:34:58.413887 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:35:08.528 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:35:08.526392 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:35:08.541 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:35:08.540325 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:35:18.603 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:35:18.600847 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:35:18.616 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:35:18.614340 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:35:28.725 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:35:28.723777 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:35:28.738 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:35:28.736830 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:35:38.822 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:35:38.820072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:35:38.837 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:35:38.835939 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:35:48.915 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:35:48.913461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:35:48.970 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:35:48.968034 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:35:59.049 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:35:59.046962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:35:59.062 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:35:59.060503 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:36:09.154 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:36:09.152263 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:36:09.167 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:36:09.165858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:36:19.291 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:36:19.288899 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:36:19.303 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:36:19.301814 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:36:29.445 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:36:29.443386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:36:29.462 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:36:29.460189 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:36:39.607 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:36:39.604844 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:36:39.618 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:36:39.616972 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:36:49.681 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:36:49.679297 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:36:49.693 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:36:49.691538 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:36:59.775 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:36:59.773005 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:36:59.843 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:36:59.841312 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:37:10.027 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:37:10.024957 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:37:10.042 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:37:10.040932 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:37:20.103 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:37:20.100439 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:37:20.116 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:37:20.115024 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:37:30.264 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:37:30.262165 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:37:30.278 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:37:30.276148 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:37:40.349 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:37:40.346961 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:37:40.361 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:37:40.359658 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:37:50.443 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:37:50.441240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:37:50.457 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:37:50.455720 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:38:00.543 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:38:00.540927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:38:00.556 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:38:00.555070 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:38:10.659 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:38:10.657623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:38:10.680 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:38:10.679288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:38:20.783 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:38:20.781731 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:38:20.796 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:38:20.794803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:38:30.910 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:38:30.908226 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:38:30.924 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:38:30.922452 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:38:41.004 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:38:41.002695 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:38:41.016 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:38:41.014533 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:38:51.090 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:38:51.088121 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:38:51.102 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:38:51.101003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:39:01.199 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:39:01.197523 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:39:01.212 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:39:01.210313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:39:11.322 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:39:11.319975 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:39:11.983 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:39:11.333344 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:39:21.417 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:39:21.414687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:39:21.455 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:39:21.453115 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:39:31.532 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:39:31.530717 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:39:31.545 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:39:31.544375 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:39:41.642 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:39:41.640748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:39:41.978 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:39:41.654096 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:39:51.732 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:39:51.730344 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:39:51.744 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:39:51.743023 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:40:01.856 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:40:01.854002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:40:01.869 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:40:01.868168 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:40:11.971 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:40:11.969516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:40:11.985 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:40:11.983976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:40:22.061 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:40:22.058971 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:40:22.073 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:40:22.072226 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:40:32.157 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:40:32.155074 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:40:32.171 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:40:32.169431 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:40:42.248 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:40:42.245857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:40:42.299 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:40:42.297567 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:40:52.397 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:40:52.395003 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:40:52.411 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:40:52.409493 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:41:02.482 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:41:02.480383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:41:02.492 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:41:02.491011 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:41:12.569 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:41:12.567017 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:41:12.584 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:41:12.582915 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:41:22.662 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:41:22.660135 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:41:22.727 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:41:22.725518 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:41:32.833 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:41:32.831007 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:41:32.846 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:41:32.844267 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:41:42.980 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:41:42.978578 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:41:42.994 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:41:42.992450 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:41:53.082 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:41:53.080199 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:41:53.095 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:41:53.093522 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:42:03.212 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:42:03.209872 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:42:03.226 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:42:03.225010 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:42:13.303 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:42:13.301779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:42:13.319 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:42:13.317319 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:42:23.439 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:42:23.437085 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:42:23.452 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:42:23.451255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:42:33.607 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:42:33.605893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:42:33.621 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:42:33.619906 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:42:43.717 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:42:43.715443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:42:43.731 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:42:43.729815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:42:53.848 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:42:53.846832 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:42:53.864 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:42:53.862906 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:43:03.944 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:43:03.942352 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:43:03.957 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:43:03.955911 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:43:14.048 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:43:14.046675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:43:14.061 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:43:14.059074 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:43:24.161 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:43:24.159407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:43:24.182 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:43:24.174559 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:43:34.287 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:43:34.284668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:43:34.301 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:43:34.299508 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:43:44.394 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:43:44.392106 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:43:44.408 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:43:44.406400 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:43:54.505 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:43:54.503212 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:43:54.517 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:43:54.516099 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:44:04.607 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:44:04.605110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:44:04.621 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:44:04.619245 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:44:14.687 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:44:14.685527 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:44:14.699 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:44:14.697323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:44:24.814 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:44:24.812122 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:44:24.823 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:44:24.821936 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:44:32.121 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:44:32.119575 I | op-mon: parsing mon endpoints: b=10.109.19.50:6789,c=10.106.65.43:6789,a=10.98.115.144:6789 2022-03-20T18:44:32.836 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:44:32.834339 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-20T18:44:32.826+0000 7f72f61c4380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-20T18:44:32.837 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T18:44:32.826+0000 7f72f61c4380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-20T18:44:32.837 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20T18:44:32.826+0000 7f72f61c4380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-20T18:44:32.837 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-20T18:44:34.892 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:44:34.890700 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:44:34.907 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:44:34.905783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:44:44.974 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:44:44.972634 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:44:44.987 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:44:44.986051 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:44:55.049 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:44:55.047598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:44:55.058 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:44:55.056794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:45:05.121 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:45:05.119519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:45:05.135 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:45:05.133224 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:45:15.267 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:45:15.265384 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:45:15.279 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:45:15.277751 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:45:25.408 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:45:25.406782 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:45:25.421 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:45:25.419404 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:45:35.535 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:45:35.533393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:45:35.546 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:45:35.544678 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:45:45.618 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:45:45.616341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:45:45.632 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:45:45.630711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:45:55.729 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:45:55.727072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:45:55.743 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:45:55.742091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:46:05.828 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:46:05.826950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:46:05.840 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:46:05.839038 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:46:15.916 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:46:15.914567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:46:15.928 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:46:15.926728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:46:26.052 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:46:26.050434 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:46:26.062 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:46:26.060896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:46:36.139 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:46:36.136965 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:46:36.154 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:46:36.152535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:46:46.273 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:46:46.271802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:46:46.287 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:46:46.285788 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:46:56.400 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:46:56.398430 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:46:56.413 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:46:56.411990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:47:06.531 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:47:06.529225 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:47:06.543 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:47:06.542233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:47:16.660 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:47:16.658632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:47:16.670 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:47:16.668355 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:47:26.800 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:47:26.797939 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:47:26.811 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:47:26.809781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:47:36.932 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:47:36.930634 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:47:36.946 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:47:36.945078 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:47:47.051 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:47:47.048853 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:47:47.065 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:47:47.063736 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:47:57.164 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:47:57.162437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:47:57.177 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:47:57.175640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:48:07.275 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:48:07.273616 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:48:07.291 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:48:07.289268 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:48:17.420 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:48:17.418201 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:48:17.434 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:48:17.432758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:48:27.569 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:48:27.567079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:48:27.579 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:48:27.577973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:48:37.768 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:48:37.766308 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:48:37.782 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:48:37.780638 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:48:47.839 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:48:47.837914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:48:47.852 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:48:47.850600 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:48:57.915 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:48:57.913894 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:48:57.990 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:48:57.988835 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:49:08.139 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:49:08.137647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:49:08.156 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:49:08.154439 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:49:18.291 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:49:18.288952 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:49:18.303 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:49:18.301995 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:49:28.453 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:49:28.451245 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:49:28.464 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:49:28.462440 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:49:38.575 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:49:38.573245 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:49:38.596 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:49:38.589521 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:49:48.651 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:49:48.645909 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:49:48.659 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:49:48.658079 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:49:58.760 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:49:58.758444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:49:58.775 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:49:58.773145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:50:08.841 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:50:08.840030 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:50:08.853 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:50:08.851498 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:50:18.923 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:50:18.921045 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:50:18.935 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:50:18.933984 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:50:29.042 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:50:29.040619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:50:29.055 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:50:29.053350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:50:39.170 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:50:39.168431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:50:39.185 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:50:39.183566 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:50:49.281 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:50:49.279351 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:50:49.292 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:50:49.290773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:50:59.431 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:50:59.429413 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:50:59.444 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:50:59.442827 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:51:09.549 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:51:09.547438 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:51:09.564 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:51:09.562806 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:51:19.676 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:51:19.673878 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:51:19.687 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:51:19.686198 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:51:29.792 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:51:29.789792 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:51:29.805 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:51:29.803595 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:51:39.881 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:51:39.879421 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:51:39.896 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:51:39.894970 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:51:49.974 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:51:49.972471 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:51:50.054 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:51:50.051941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:52:00.168 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:52:00.166382 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:52:00.182 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:52:00.180428 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:52:10.277 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:52:10.274961 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:52:10.290 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:52:10.288327 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:52:20.357 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:52:20.355540 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:52:20.370 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:52:20.368547 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:52:30.461 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:52:30.459573 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:52:30.474 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:52:30.472790 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:52:40.644 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:52:40.642835 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:52:40.659 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:52:40.658096 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:52:50.803 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:52:50.801625 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T18:52:50.817 INFO:tasks.rook.operator.smithi007.stdout:2022-03-20 18:52:50.815401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-20T21:13:50.417 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-03-20T21:13:50.446 DEBUG:teuthology.task.console_log:Killing console logger for smithi007 2022-03-20T21:13:50.447 DEBUG:teuthology.task.console_log:Killing console logger for smithi057 2022-03-20T21:13:50.448 DEBUG:teuthology.task.console_log:Killing console logger for smithi087 2022-03-20T21:13:50.449 DEBUG:teuthology.exit:Finished running handlers