2022-04-09T11:27:34.377 INFO:root:teuthology version: 1.1.0-a347b2ac 2022-04-09T11:27:34.379 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/ozeneva-2022-04-09_11:21:29-rados-wip-omri-tracing-compiled-distro-basic-smithi/6783799 --verbose --description rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench 3-final cluster/1-node k8s/1.21 net/flannel rook/1.7.2} --owner scheduled_ozeneva@teuthology --name ozeneva-2022-04-09_11:21:29-rados-wip-omri-tracing-compiled-distro-basic-smithi -- /home/teuthworker/archive/ozeneva-2022-04-09_11:21:29-rados-wip-omri-tracing-compiled-distro-basic-smithi/6783799/orig.config.yaml 2022-04-09T11:27:34.412 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-04-09T11:27:34.475 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/ozeneva-2022-04-09_11:21:29-rados-wip-omri-tracing-compiled-distro-basic-smithi/6783799 branch: wip-omri-tracing-compiled description: rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench 3-final cluster/1-node k8s/1.21 net/flannel rook/1.7.2} email: null first_in_suite: false job_id: '6783799' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: ozeneva-2022-04-09_11:21:29-rados-wip-omri-tracing-compiled-distro-basic-smithi nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-omri-tracing-compiled ceph: conf: global: osd crush chooseleaf type: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: a70e525ea98f7194c5e516757c6105ca14c81cb7 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: a70e525ea98f7194c5e516757c6105ca14c81cb7 kernel: hwe: true kubeadm: pod_network: flannel version: '1.21' rook: rook_branch: v1.7.2 rook_image: rook/ceph:v1.7.2 workunit: branch: wip-omri-tracing-compiled sha1: a70e525ea98f7194c5e516757c6105ca14c81cb7 owner: scheduled_ozeneva@teuthology priority: 50 repo: git://git.ceph.com/ceph-ci.git roles: - - host.a - client.a sha1: a70e525ea98f7194c5e516757c6105ca14c81cb7 sleep_before_teardown: 0 suite: rados suite_branch: wip-omri-tracing-compiled suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_a70e525ea98f7194c5e516757c6105ca14c81cb7/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: a70e525ea98f7194c5e516757c6105ca14c81cb7 targets: smithi119.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBRq6WrB1GWg4PrKvCBN9Vq8H1+8wAoRqMXdH9n4moPqas2+WRzSagQxMUMgKIJ8BDOuxVO3Fj6NlZUNbxzhaNJvX5/QhdIq75Zo6L8YyGpth6qn6SwhqUXhgwdjUZ0gltmbS6vEwQB6WvwjZLDusyqpfcGtfpGNnb2WNLD5c2j7/lGEGQ7Tf+TDdiyKzx+jbvhAWJyIffDmUpMdt/dohTdDjlgDgZ0UBh99g25HTAa2MYEdtyx9c6ZaCKAQ3Qo8aGxs98sPkkgEB0APyUVGTXSbLmtuDcdVn0f/WSewOUFt3nb8FOGGsFemxiSf8D5rvRTR1bli+oNezFKCAtpF6GnKkZiZszZzgdTvMfP+I3nZdJm5QPLc+K5Hi+GwkS6rF46VVWVh5+PfzOVT+rUl+4A/FAlRQgTEz9vSVHVdIhX/Bgmjy6CIiFL6rf1PwaCfnv7YyQ18NnPz5OWNY9+5OPOPfMx49V+BWTfr/2p2cqBPP9pG6zW8lY+kNbYF6Yzm8= tasks: - kubeadm: null - nvme_loop: null - rook: null - rook.shell: - ceph -s - ceph orch status - ceph orch ps - ceph orch ls - ceph orch device ls - install: host.a: null - radosbench: clients: - client.a - rook.shell: commands: - 'ceph orch host label add `hostname` foo ceph orch host ls | grep foo ceph orch host label rm `hostname` foo ceph orch host ls | grep -v foo ' - exec: host.a: - "set -ex\ntoolbox() {\n kubectl -n rook-ceph exec -it deploy/rook-ceph-tools\ \ -- \"$@\"\n}\norig_num_osd=`toolbox ceph osd stat | cut -f3 -d \" \"`\ntoolbox\ \ ceph orch osd rm 0 --force\nremoved_pv=\"\"\nwhile [ \"$removed_pv\" = \"\"\ \ ]\ndo\n removed_pv=`kubectl get pv | grep Released | cut -f1 -d \" \"`\n\ \ sleep 3s\ndone\ntarget_path=`kubectl get pv $removed_pv -o jsonpath='{.spec.local.path}'`\n\ host=`echo $removed_pv | cut -f1 -d \"-\"`\ntoolbox ceph orch device zap $host\ \ $target_path --force\nzap_completion=\"0\"\nwhile [ \"$zap_completion\" =\ \ \"0\" ]\ndo\n zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap\ \ -o jsonpath='{.status.succeeded.path}'`\n sleep 3s\ndone\nkubectl patch\ \ pv $removed_pv -p '{\"spec\":{\"claimRef\": null}}'\ntoolbox ceph orch apply\ \ osd --all-available-devices\nkubectl delete job rook-ceph-device-zap -n rook-ceph\n\ num_osd=\"0\"\nwhile [ \"$num_osd\" != \"$orig_num_osd\" ]\ndo\n echo \"\ waiting for osd to come back up\"\n num_osd=`toolbox ceph osd stat | cut\ \ -f3 -d \" \"`\n sleep 30s\ndone\n" - rook.shell: commands: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch apply rgw foo - ceph orch apply mds foo - ceph orch apply rbd-mirror - ceph orch apply nfs foo --port 12777 teuthology_branch: master teuthology_sha1: a347b2ac5400a43ba76a678399b1deda7dae1a6c timestamp: 2022-04-09_11:21:29 tube: smithi user: ozeneva verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.22301 2022-04-09T11:27:34.476 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_a70e525ea98f7194c5e516757c6105ca14c81cb7/qa; will attempt to use it 2022-04-09T11:27:34.477 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_a70e525ea98f7194c5e516757c6105ca14c81cb7/qa/tasks 2022-04-09T11:27:34.477 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-04-09T11:27:34.479 INFO:teuthology.task.internal:Checking packages... 2022-04-09T11:27:34.501 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'a70e525ea98f7194c5e516757c6105ca14c81cb7' 2022-04-09T11:27:34.501 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-04-09T11:27:34.502 INFO:teuthology.packaging:ref: None 2022-04-09T11:27:34.502 INFO:teuthology.packaging:tag: None 2022-04-09T11:27:34.502 INFO:teuthology.packaging:branch: wip-omri-tracing-compiled 2022-04-09T11:27:34.502 INFO:teuthology.packaging:sha1: a70e525ea98f7194c5e516757c6105ca14c81cb7 2022-04-09T11:27:34.503 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-omri-tracing-compiled 2022-04-09T11:27:34.662 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-11488-ga70e525e-1focal 2022-04-09T11:27:34.662 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-04-09T11:27:34.673 INFO:teuthology.task.internal:no buildpackages task found 2022-04-09T11:27:34.674 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-04-09T11:27:34.744 INFO:teuthology.task.internal:Saving configuration 2022-04-09T11:27:34.757 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-04-09T11:27:34.793 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-04-09T11:27:34.816 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi119.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/ozeneva-2022-04-09_11:21:29-rados-wip-omri-tracing-compiled-distro-basic-smithi/6783799', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-04-09 11:22:22.606855', 'locked_by': 'scheduled_ozeneva@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINRDdPUMMdlUoC2YcEeoO0xR2GLus0CYVzGGXJxLzVna'} 2022-04-09T11:27:34.817 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-04-09T11:27:34.901 INFO:teuthology.task.internal:roles: ubuntu@smithi119.front.sepia.ceph.com - ['host.a', 'client.a'] 2022-04-09T11:27:34.901 INFO:teuthology.run_tasks:Running task console_log... 2022-04-09T11:27:35.041 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f210cc12950>, signals=[15]) 2022-04-09T11:27:35.042 INFO:teuthology.run_tasks:Running task internal.connect... 2022-04-09T11:27:35.096 INFO:teuthology.task.internal:Opening connections... 2022-04-09T11:27:35.096 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi119.front.sepia.ceph.com 2022-04-09T11:27:35.098 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-09T11:27:35.148 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-04-09T11:27:35.188 DEBUG:teuthology.orchestra.run.smithi119:> uname -m 2022-04-09T11:27:35.277 INFO:teuthology.orchestra.run.smithi119.stdout:x86_64 2022-04-09T11:27:35.277 DEBUG:teuthology.orchestra.run.smithi119:> cat /etc/os-release 2022-04-09T11:27:35.325 INFO:teuthology.orchestra.run.smithi119.stdout:NAME="Ubuntu" 2022-04-09T11:27:35.325 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-04-09T11:27:35.325 INFO:teuthology.orchestra.run.smithi119.stdout:ID=ubuntu 2022-04-09T11:27:35.326 INFO:teuthology.orchestra.run.smithi119.stdout:ID_LIKE=debian 2022-04-09T11:27:35.326 INFO:teuthology.orchestra.run.smithi119.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-04-09T11:27:35.326 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION_ID="20.04" 2022-04-09T11:27:35.326 INFO:teuthology.orchestra.run.smithi119.stdout:HOME_URL="https://www.ubuntu.com/" 2022-04-09T11:27:35.327 INFO:teuthology.orchestra.run.smithi119.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-04-09T11:27:35.327 INFO:teuthology.orchestra.run.smithi119.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-04-09T11:27:35.327 INFO:teuthology.orchestra.run.smithi119.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-04-09T11:27:35.327 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION_CODENAME=focal 2022-04-09T11:27:35.328 INFO:teuthology.orchestra.run.smithi119.stdout:UBUNTU_CODENAME=focal 2022-04-09T11:27:35.328 INFO:teuthology.lock.ops:Updating smithi119.front.sepia.ceph.com on lock server 2022-04-09T11:27:35.347 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-04-09T11:27:35.360 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-04-09T11:27:35.432 INFO:teuthology.task.internal:Checking for old test directory... 2022-04-09T11:27:35.432 DEBUG:teuthology.orchestra.run.smithi119:> test '!' -e /home/ubuntu/cephtest 2022-04-09T11:27:35.439 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-04-09T11:27:35.526 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-04-09T11:27:35.527 DEBUG:teuthology.orchestra.run.smithi119:> test -z $(ls -A /var/lib/ceph) 2022-04-09T11:27:35.536 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-04-09T11:27:35.649 INFO:teuthology.run_tasks:Running task kernel... 2022-04-09T11:27:35.741 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-04-09T11:27:35.742 INFO:teuthology.task.kernel:normalize config orig: {'hwe': True} 2022-04-09T11:27:35.742 DEBUG:teuthology.task.kernel:normalized overrides {'host.a': {'hwe': True}, 'client.a': {'hwe': True}} 2022-04-09T11:27:35.743 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro', 'hwe': True}}, timeout 300 2022-04-09T11:27:35.743 DEBUG:teuthology.orchestra.run.smithi119:> test -f /run/.containerenv -o -f /.dockerenv 2022-04-09T11:27:35.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-09T11:27:35.750 DEBUG:teuthology.orchestra.run.smithi119:> uname -r 2022-04-09T11:27:35.757 INFO:teuthology.orchestra.run.smithi119.stdout:5.4.0-107-generic 2022-04-09T11:27:35.757 INFO:teuthology.task.kernel:Running kernel on smithi119: 5.4.0-107-generic 2022-04-09T11:27:35.758 DEBUG:teuthology.orchestra.run.smithi119:> sudo apt-get clean 2022-04-09T11:27:35.850 DEBUG:teuthology.orchestra.run.smithi119:> sudo apt-get update 2022-04-09T11:27:36.009 INFO:teuthology.orchestra.run.smithi119.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-04-09T11:27:36.110 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-04-09T11:27:36.191 INFO:teuthology.orchestra.run.smithi119.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-04-09T11:27:36.350 INFO:teuthology.orchestra.run.smithi119.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,384 kB] 2022-04-09T11:27:36.463 INFO:teuthology.orchestra.run.smithi119.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [417 kB] 2022-04-09T11:27:36.478 INFO:teuthology.orchestra.run.smithi119.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [239 kB] 2022-04-09T11:27:36.479 INFO:teuthology.orchestra.run.smithi119.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,884 B] 2022-04-09T11:27:36.479 INFO:teuthology.orchestra.run.smithi119.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [859 kB] 2022-04-09T11:27:36.483 INFO:teuthology.orchestra.run.smithi119.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [122 kB] 2022-04-09T11:27:36.485 INFO:teuthology.orchestra.run.smithi119.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [547 kB] 2022-04-09T11:27:36.492 INFO:teuthology.orchestra.run.smithi119.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [696 kB] 2022-04-09T11:27:36.501 INFO:teuthology.orchestra.run.smithi119.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [122 kB] 2022-04-09T11:27:36.502 INFO:teuthology.orchestra.run.smithi119.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.1 kB] 2022-04-09T11:27:36.537 INFO:teuthology.orchestra.run.smithi119.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-04-09T11:27:36.686 INFO:teuthology.orchestra.run.smithi119.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,712 kB] 2022-04-09T11:27:37.079 INFO:teuthology.orchestra.run.smithi119.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [629 kB] 2022-04-09T11:27:37.098 INFO:teuthology.orchestra.run.smithi119.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [320 kB] 2022-04-09T11:27:37.108 INFO:teuthology.orchestra.run.smithi119.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.9 kB] 2022-04-09T11:27:37.109 INFO:teuthology.orchestra.run.smithi119.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [917 kB] 2022-04-09T11:27:37.161 INFO:teuthology.orchestra.run.smithi119.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [131 kB] 2022-04-09T11:27:37.165 INFO:teuthology.orchestra.run.smithi119.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [916 kB] 2022-04-09T11:27:37.194 INFO:teuthology.orchestra.run.smithi119.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [676 kB] 2022-04-09T11:27:37.214 INFO:teuthology.orchestra.run.smithi119.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [204 kB] 2022-04-09T11:27:37.219 INFO:teuthology.orchestra.run.smithi119.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.4 kB] 2022-04-09T11:27:40.035 INFO:teuthology.orchestra.run.smithi119.stdout:Fetched 10.3 MB in 2s (6,167 kB/s) 2022-04-09T11:27:41.254 INFO:teuthology.orchestra.run.smithi119.stdout:Reading package lists... 2022-04-09T11:27:41.278 DEBUG:teuthology.orchestra.run.smithi119:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-04-09T11:27:41.348 INFO:teuthology.orchestra.run.smithi119.stdout:Reading package lists... 2022-04-09T11:27:41.524 INFO:teuthology.orchestra.run.smithi119.stdout:Building dependency tree... 2022-04-09T11:27:41.525 INFO:teuthology.orchestra.run.smithi119.stdout:Reading state information... 2022-04-09T11:27:41.671 INFO:teuthology.orchestra.run.smithi119.stdout:The following additional packages will be installed: 2022-04-09T11:27:41.671 INFO:teuthology.orchestra.run.smithi119.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-04-09T11:27:41.672 INFO:teuthology.orchestra.run.smithi119.stdout: libusbmuxd6 linux-image-5.13.0-39-generic linux-modules-5.13.0-39-generic 2022-04-09T11:27:41.672 INFO:teuthology.orchestra.run.smithi119.stdout: linux-modules-extra-5.13.0-39-generic thermald upower usbmuxd 2022-04-09T11:27:41.672 INFO:teuthology.orchestra.run.smithi119.stdout:Suggested packages: 2022-04-09T11:27:41.673 INFO:teuthology.orchestra.run.smithi119.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-04-09T11:27:41.673 INFO:teuthology.orchestra.run.smithi119.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-39-generic 2022-04-09T11:27:41.722 INFO:teuthology.orchestra.run.smithi119.stdout:The following NEW packages will be installed: 2022-04-09T11:27:41.723 INFO:teuthology.orchestra.run.smithi119.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-04-09T11:27:41.723 INFO:teuthology.orchestra.run.smithi119.stdout: libusbmuxd6 linux-image-5.13.0-39-generic linux-image-generic-hwe-20.04 2022-04-09T11:27:41.724 INFO:teuthology.orchestra.run.smithi119.stdout: linux-modules-5.13.0-39-generic linux-modules-extra-5.13.0-39-generic 2022-04-09T11:27:41.724 INFO:teuthology.orchestra.run.smithi119.stdout: thermald upower usbmuxd 2022-04-09T11:27:41.783 INFO:teuthology.orchestra.run.smithi119.stdout:0 upgraded, 13 newly installed, 0 to remove and 8 not upgraded. 2022-04-09T11:27:41.784 INFO:teuthology.orchestra.run.smithi119.stdout:Need to get 79.9 MB of archives. 2022-04-09T11:27:41.784 INFO:teuthology.orchestra.run.smithi119.stdout:After this operation, 410 MB of additional disk space will be used. 2022-04-09T11:27:41.784 INFO:teuthology.orchestra.run.smithi119.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-04-09T11:27:41.852 INFO:teuthology.orchestra.run.smithi119.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-04-09T11:27:41.856 INFO:teuthology.orchestra.run.smithi119.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-04-09T11:27:41.857 INFO:teuthology.orchestra.run.smithi119.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-04-09T11:27:41.880 INFO:teuthology.orchestra.run.smithi119.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-04-09T11:27:41.893 INFO:teuthology.orchestra.run.smithi119.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-39-generic amd64 5.13.0-39.44~20.04.1 [18.3 MB] 2022-04-09T11:27:42.341 INFO:teuthology.orchestra.run.smithi119.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-39-generic amd64 5.13.0-39.44~20.04.1 [10.0 MB] 2022-04-09T11:27:42.463 INFO:teuthology.orchestra.run.smithi119.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-39-generic amd64 5.13.0-39.44~20.04.1 [51.0 MB] 2022-04-09T11:27:43.552 INFO:teuthology.orchestra.run.smithi119.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.39.44~20.04.24 [2,620 B] 2022-04-09T11:27:43.633 INFO:teuthology.orchestra.run.smithi119.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-04-09T11:27:43.829 INFO:teuthology.orchestra.run.smithi119.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-04-09T11:27:44.140 INFO:teuthology.orchestra.run.smithi119.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-04-09T11:27:44.242 INFO:teuthology.orchestra.run.smithi119.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-04-09T11:27:44.683 INFO:teuthology.orchestra.run.smithi119.stdout:Fetched 79.9 MB in 3s (31.7 MB/s) 2022-04-09T11:27:44.856 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-04-09T11:27:46.387 INFO:teuthology.orchestra.run.smithi119.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102172 files and directories currently installed.) 2022-04-09T11:27:46.391 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-04-09T11:27:46.536 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-04-09T11:27:46.882 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libplist3:amd64. 2022-04-09T11:27:46.895 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-04-09T11:27:46.931 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-04-09T11:27:47.242 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-04-09T11:27:47.257 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-04-09T11:27:47.325 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-04-09T11:27:47.653 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-04-09T11:27:47.666 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-04-09T11:27:47.703 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-04-09T11:27:48.014 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-04-09T11:27:48.029 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-04-09T11:27:48.064 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-04-09T11:27:48.401 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package linux-modules-5.13.0-39-generic. 2022-04-09T11:27:48.416 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../05-linux-modules-5.13.0-39-generic_5.13.0-39.44~20.04.1_amd64.deb ... 2022-04-09T11:27:48.459 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking linux-modules-5.13.0-39-generic (5.13.0-39.44~20.04.1) ... 2022-04-09T11:27:50.693 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package linux-image-5.13.0-39-generic. 2022-04-09T11:27:50.706 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../06-linux-image-5.13.0-39-generic_5.13.0-39.44~20.04.1_amd64.deb ... 2022-04-09T11:27:50.812 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking linux-image-5.13.0-39-generic (5.13.0-39.44~20.04.1) ... 2022-04-09T11:27:51.313 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-39-generic. 2022-04-09T11:27:51.326 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-39-generic_5.13.0-39.44~20.04.1_amd64.deb ... 2022-04-09T11:27:51.363 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking linux-modules-extra-5.13.0-39-generic (5.13.0-39.44~20.04.1) ... 2022-04-09T11:27:57.672 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-04-09T11:27:57.686 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.39.44~20.04.24_amd64.deb ... 2022-04-09T11:27:57.721 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.39.44~20.04.24) ... 2022-04-09T11:27:58.049 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libevdev2:amd64. 2022-04-09T11:27:58.065 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-04-09T11:27:58.107 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-04-09T11:27:58.485 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package thermald. 2022-04-09T11:27:58.499 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-04-09T11:27:58.543 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-04-09T11:27:59.065 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package upower. 2022-04-09T11:27:59.079 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-04-09T11:27:59.133 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking upower (0.99.11-1build2) ... 2022-04-09T11:27:59.561 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package usbmuxd. 2022-04-09T11:27:59.575 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-04-09T11:27:59.619 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-04-09T11:27:59.927 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-04-09T11:28:00.040 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-04-09T11:28:00.175 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-04-09T11:28:00.300 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-04-09T11:28:00.427 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-04-09T11:28:00.553 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-04-09T11:28:00.695 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up upower (0.99.11-1build2) ... 2022-04-09T11:28:01.130 INFO:teuthology.orchestra.run.smithi119.stdout:upower.service is a disabled or a static unit, not starting it. 2022-04-09T11:28:01.182 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-04-09T11:28:01.378 INFO:teuthology.orchestra.run.smithi119.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-04-09T11:28:01.387 INFO:teuthology.orchestra.run.smithi119.stdout:Adding system user `usbmux' (UID 114) ... 2022-04-09T11:28:01.387 INFO:teuthology.orchestra.run.smithi119.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-04-09T11:28:01.882 INFO:teuthology.orchestra.run.smithi119.stdout:Not creating home directory `/var/lib/usbmux'. 2022-04-09T11:28:02.117 INFO:teuthology.orchestra.run.smithi119.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-04-09T11:28:02.153 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-04-09T11:28:02.467 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-04-09T11:28:02.467 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-04-09T11:28:02.957 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up linux-modules-5.13.0-39-generic (5.13.0-39.44~20.04.1) ... 2022-04-09T11:28:04.310 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up linux-image-5.13.0-39-generic (5.13.0-39.44~20.04.1) ... 2022-04-09T11:28:05.584 INFO:teuthology.orchestra.run.smithi119.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-39-generic 2022-04-09T11:28:05.584 INFO:teuthology.orchestra.run.smithi119.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-39-generic 2022-04-09T11:28:05.728 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up linux-modules-extra-5.13.0-39-generic (5.13.0-39.44~20.04.1) ... 2022-04-09T11:28:07.054 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.39.44~20.04.24) ... 2022-04-09T11:28:07.188 INFO:teuthology.orchestra.run.smithi119.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.7) ... 2022-04-09T11:28:11.652 INFO:teuthology.orchestra.run.smithi119.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-04-09T11:28:12.270 INFO:teuthology.orchestra.run.smithi119.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-04-09T11:28:12.371 INFO:teuthology.orchestra.run.smithi119.stdout:Processing triggers for linux-image-5.13.0-39-generic (5.13.0-39.44~20.04.1) ... 2022-04-09T11:28:12.574 INFO:teuthology.orchestra.run.smithi119.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-04-09T11:28:12.574 INFO:teuthology.orchestra.run.smithi119.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-39-generic 2022-04-09T11:28:36.335 INFO:teuthology.orchestra.run.smithi119.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-04-09T11:28:36.335 INFO:teuthology.orchestra.run.smithi119.stdout:Sourcing file `/etc/default/grub' 2022-04-09T11:28:36.354 INFO:teuthology.orchestra.run.smithi119.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-04-09T11:28:36.372 INFO:teuthology.orchestra.run.smithi119.stdout:Generating grub configuration file ... 2022-04-09T11:28:36.667 INFO:teuthology.orchestra.run.smithi119.stdout:Found linux image: /boot/vmlinuz-5.13.0-39-generic 2022-04-09T11:28:36.690 INFO:teuthology.orchestra.run.smithi119.stdout:Found initrd image: /boot/initrd.img-5.13.0-39-generic 2022-04-09T11:28:37.098 INFO:teuthology.orchestra.run.smithi119.stdout:Found linux image: /boot/vmlinuz-5.4.0-107-generic 2022-04-09T11:28:37.110 INFO:teuthology.orchestra.run.smithi119.stdout:Found initrd image: /boot/initrd.img-5.4.0-107-generic 2022-04-09T11:28:39.537 INFO:teuthology.orchestra.run.smithi119.stdout:done 2022-04-09T11:28:39.919 DEBUG:teuthology.orchestra.run.smithi119:> dpkg -s linux-image-generic-hwe-20.04 2022-04-09T11:28:39.950 INFO:teuthology.orchestra.run.smithi119.stdout:Package: linux-image-generic-hwe-20.04 2022-04-09T11:28:39.950 INFO:teuthology.orchestra.run.smithi119.stdout:Status: install ok installed 2022-04-09T11:28:39.951 INFO:teuthology.orchestra.run.smithi119.stdout:Priority: optional 2022-04-09T11:28:39.951 INFO:teuthology.orchestra.run.smithi119.stdout:Section: kernel 2022-04-09T11:28:39.951 INFO:teuthology.orchestra.run.smithi119.stdout:Installed-Size: 19 2022-04-09T11:28:39.951 INFO:teuthology.orchestra.run.smithi119.stdout:Maintainer: Ubuntu Kernel Team 2022-04-09T11:28:39.952 INFO:teuthology.orchestra.run.smithi119.stdout:Architecture: amd64 2022-04-09T11:28:39.952 INFO:teuthology.orchestra.run.smithi119.stdout:Source: linux-meta-hwe-5.13 2022-04-09T11:28:39.952 INFO:teuthology.orchestra.run.smithi119.stdout:Version: 5.13.0.39.44~20.04.24 2022-04-09T11:28:39.952 INFO:teuthology.orchestra.run.smithi119.stdout:Provides: virtualbox-guest-modules (= 5.13.0-39), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2.1) 2022-04-09T11:28:39.952 INFO:teuthology.orchestra.run.smithi119.stdout:Depends: linux-image-5.13.0-39-generic, linux-modules-extra-5.13.0-39-generic, linux-firmware, intel-microcode, amd64-microcode 2022-04-09T11:28:39.953 INFO:teuthology.orchestra.run.smithi119.stdout:Recommends: thermald 2022-04-09T11:28:39.953 INFO:teuthology.orchestra.run.smithi119.stdout:Description: Generic Linux kernel image 2022-04-09T11:28:39.953 INFO:teuthology.orchestra.run.smithi119.stdout: This package will always depend on the latest generic kernel image 2022-04-09T11:28:39.953 INFO:teuthology.orchestra.run.smithi119.stdout: available. 2022-04-09T11:28:39.954 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-107-generic Expected: 5.13.0-39-generic 2022-04-09T11:28:39.954 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-04-09T11:28:39.954 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-04-09T11:28:39.955 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2022-04-09T11:28:39.955 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi119.front.sepia.ceph.com, path=None, version=distro) 2022-04-09T11:28:39.955 DEBUG:teuthology.orchestra.run.smithi119:> sudo apt-get clean 2022-04-09T11:28:40.015 DEBUG:teuthology.orchestra.run.smithi119:> sudo apt-get update 2022-04-09T11:28:40.168 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-04-09T11:28:40.183 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-04-09T11:28:40.203 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-04-09T11:28:40.275 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-04-09T11:28:41.701 INFO:teuthology.orchestra.run.smithi119.stdout:Reading package lists... 2022-04-09T11:28:41.724 DEBUG:teuthology.orchestra.run.smithi119:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-04-09T11:28:41.792 INFO:teuthology.orchestra.run.smithi119.stdout:Reading package lists... 2022-04-09T11:28:41.968 INFO:teuthology.orchestra.run.smithi119.stdout:Building dependency tree... 2022-04-09T11:28:41.969 INFO:teuthology.orchestra.run.smithi119.stdout:Reading state information... 2022-04-09T11:28:42.146 INFO:teuthology.orchestra.run.smithi119.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.39.44~20.04.24). 2022-04-09T11:28:42.147 INFO:teuthology.orchestra.run.smithi119.stdout:0 upgraded, 0 newly installed, 0 to remove and 8 not upgraded. 2022-04-09T11:28:42.148 DEBUG:teuthology.orchestra.run.smithi119:> dpkg -s linux-image-generic-hwe-20.04 2022-04-09T11:28:42.171 INFO:teuthology.orchestra.run.smithi119.stdout:Package: linux-image-generic-hwe-20.04 2022-04-09T11:28:42.171 INFO:teuthology.orchestra.run.smithi119.stdout:Status: install ok installed 2022-04-09T11:28:42.171 INFO:teuthology.orchestra.run.smithi119.stdout:Priority: optional 2022-04-09T11:28:42.171 INFO:teuthology.orchestra.run.smithi119.stdout:Section: kernel 2022-04-09T11:28:42.172 INFO:teuthology.orchestra.run.smithi119.stdout:Installed-Size: 19 2022-04-09T11:28:42.172 INFO:teuthology.orchestra.run.smithi119.stdout:Maintainer: Ubuntu Kernel Team 2022-04-09T11:28:42.172 INFO:teuthology.orchestra.run.smithi119.stdout:Architecture: amd64 2022-04-09T11:28:42.172 INFO:teuthology.orchestra.run.smithi119.stdout:Source: linux-meta-hwe-5.13 2022-04-09T11:28:42.173 INFO:teuthology.orchestra.run.smithi119.stdout:Version: 5.13.0.39.44~20.04.24 2022-04-09T11:28:42.173 INFO:teuthology.orchestra.run.smithi119.stdout:Provides: virtualbox-guest-modules (= 5.13.0-39), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2.1) 2022-04-09T11:28:42.173 INFO:teuthology.orchestra.run.smithi119.stdout:Depends: linux-image-5.13.0-39-generic, linux-modules-extra-5.13.0-39-generic, linux-firmware, intel-microcode, amd64-microcode 2022-04-09T11:28:42.173 INFO:teuthology.orchestra.run.smithi119.stdout:Recommends: thermald 2022-04-09T11:28:42.173 INFO:teuthology.orchestra.run.smithi119.stdout:Description: Generic Linux kernel image 2022-04-09T11:28:42.174 INFO:teuthology.orchestra.run.smithi119.stdout: This package will always depend on the latest generic kernel image 2022-04-09T11:28:42.174 INFO:teuthology.orchestra.run.smithi119.stdout: available. 2022-04-09T11:28:42.174 DEBUG:teuthology.orchestra.run.smithi119:> mktemp 2022-04-09T11:28:42.218 INFO:teuthology.orchestra.run.smithi119.stdout:/tmp/tmp.TQNahY8rL4 2022-04-09T11:28:42.218 DEBUG:teuthology.orchestra.run.smithi119:> sudo cp /boot/grub/grub.cfg /tmp/tmp.TQNahY8rL4 2022-04-09T11:28:42.232 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0666 /tmp/tmp.TQNahY8rL4 2022-04-09T11:28:42.273 DEBUG:teuthology.orchestra.remote:smithi119:/tmp/tmp.TQNahY8rL4 is 8KB 2022-04-09T11:28:42.287 DEBUG:teuthology.orchestra.run.smithi119:> rm -fr /tmp/tmp.TQNahY8rL4 2022-04-09T11:28:42.294 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-04-09T11:28:42.349 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-04-09T11:28:42.349 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-04-09T11:28:42.350 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-04-09T11:28:42.394 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-39-generic 2022-04-09T11:28:42.394 DEBUG:teuthology.orchestra.run.smithi119:> sudo update-grub 2022-04-09T11:28:43.640 INFO:teuthology.orchestra.run.smithi119.stderr:Sourcing file `/etc/default/grub' 2022-04-09T11:28:43.644 INFO:teuthology.orchestra.run.smithi119.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-04-09T11:28:43.661 INFO:teuthology.orchestra.run.smithi119.stderr:Generating grub configuration file ... 2022-04-09T11:28:43.893 INFO:teuthology.orchestra.run.smithi119.stderr:Found linux image: /boot/vmlinuz-5.13.0-39-generic 2022-04-09T11:28:43.904 INFO:teuthology.orchestra.run.smithi119.stderr:Found initrd image: /boot/initrd.img-5.13.0-39-generic 2022-04-09T11:28:44.283 INFO:teuthology.orchestra.run.smithi119.stderr:Found linux image: /boot/vmlinuz-5.4.0-107-generic 2022-04-09T11:28:44.293 INFO:teuthology.orchestra.run.smithi119.stderr:Found initrd image: /boot/initrd.img-5.4.0-107-generic 2022-04-09T11:28:45.711 INFO:teuthology.orchestra.run.smithi119.stderr:done 2022-04-09T11:28:45.713 DEBUG:teuthology.orchestra.run.smithi119:> sudo shutdown -r now 2022-04-09T11:29:15.742 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2022-04-09T11:29:15.743 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2022-04-09T11:29:15.744 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-09T11:29:34.159 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.119 2022-04-09T11:30:04.187 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2022-04-09T11:30:04.188 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-09T11:30:20.491 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.119 2022-04-09T11:30:50.519 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2022-04-09T11:30:50.520 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-09T11:30:50.699 DEBUG:teuthology.orchestra.run.smithi119:> true 2022-04-09T11:30:51.220 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi119.front.sepia.ceph.com' 2022-04-09T11:30:51.221 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.13.0-39-generic"... 2022-04-09T11:30:51.221 DEBUG:teuthology.orchestra.run.smithi119:> uname -r 2022-04-09T11:30:51.226 INFO:teuthology.orchestra.run.smithi119.stdout:5.13.0-39-generic 2022-04-09T11:30:51.227 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-39-generic vs 5.13.0-39-generic 2022-04-09T11:30:51.227 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-04-09T11:30:51.227 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-04-09T11:30:52.229 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2022-04-09T11:30:52.229 DEBUG:teuthology.orchestra.run.smithi119:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-04-09T11:30:52.291 INFO:teuthology.orchestra.run.smithi119.stdout:ttyS1 2022-04-09T11:30:52.315 DEBUG:teuthology.parallel:result is None 2022-04-09T11:30:52.315 INFO:teuthology.run_tasks:Running task internal.base... 2022-04-09T11:30:52.324 INFO:teuthology.task.internal:Creating test directory... 2022-04-09T11:30:52.325 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-04-09T11:30:52.332 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-04-09T11:30:52.424 INFO:teuthology.run_tasks:Running task internal.archive... 2022-04-09T11:30:52.666 INFO:teuthology.task.internal:Creating archive directory... 2022-04-09T11:30:52.666 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-04-09T11:30:52.699 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-04-09T11:30:52.906 INFO:teuthology.task.internal:Enabling coredump saving... 2022-04-09T11:30:52.906 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-04-09T11:30:52.931 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-04-09T11:30:52.936 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-04-09T11:30:52.937 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-04-09T11:30:53.058 INFO:teuthology.task.internal:Configuring sudo... 2022-04-09T11:30:53.059 DEBUG:teuthology.orchestra.run.smithi119:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-04-09T11:30:53.076 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-04-09T11:30:53.203 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-04-09T11:30:53.204 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-04-09T11:30:53.211 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-04-09T11:30:53.219 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-04-09T11:30:53.227 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-04-09T11:30:53.227 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-04-09T11:30:53.253 DEBUG:teuthology.orchestra.run.smithi119:> sudo service rsyslog restart 2022-04-09T11:30:53.342 INFO:teuthology.run_tasks:Running task internal.timer... 2022-04-09T11:30:53.366 INFO:teuthology.task.internal:Starting timer... 2022-04-09T11:30:53.367 INFO:teuthology.run_tasks:Running task pcp... 2022-04-09T11:30:53.492 INFO:teuthology.run_tasks:Running task selinux... 2022-04-09T11:30:53.624 DEBUG:teuthology.task.selinux:Excluding smithi119: OS 'ubuntu' does not support SELinux 2022-04-09T11:30:53.625 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-04-09T11:30:53.625 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-04-09T11:30:53.625 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-04-09T11:30:53.626 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-04-09T11:30:53.772 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2022-04-09T11:30:53.779 INFO:teuthology.repo_utils:Fetching master from origin 2022-04-09T11:30:53.809 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2022-04-09T11:30:53.827 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-04-09T11:30:53.829 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi119.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2022-04-09T11:33:41.369 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi119.front.sepia.ceph.com')] 2022-04-09T11:33:41.371 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2022-04-09T11:33:41.373 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-09T11:33:41.418 DEBUG:teuthology.orchestra.run.smithi119:> true 2022-04-09T11:33:41.540 INFO:teuthology.run_tasks:Running task clock... 2022-04-09T11:33:41.554 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-04-09T11:33:41.555 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-04-09T11:33:41.557 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-04-09T11:33:41.594 INFO:teuthology.orchestra.run.smithi119.stdout: 9 Apr 11:33:41 ntpd[7816]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-04-09T11:33:41.595 INFO:teuthology.orchestra.run.smithi119.stdout: 9 Apr 11:33:41 ntpd[7816]: Command line: ntpd -gq 2022-04-09T11:33:41.595 INFO:teuthology.orchestra.run.smithi119.stdout: 9 Apr 11:33:41 ntpd[7816]: proto: precision = 0.116 usec (-23) 2022-04-09T11:33:41.596 INFO:teuthology.orchestra.run.smithi119.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-04-09T11:33:41.596 INFO:teuthology.orchestra.run.smithi119.stdout: 9 Apr 11:33:41 ntpd[7816]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-04-09T11:33:41.597 INFO:teuthology.orchestra.run.smithi119.stderr:restrict ::: KOD does nothing without LIMITED. 2022-04-09T11:33:41.598 INFO:teuthology.orchestra.run.smithi119.stdout: 9 Apr 11:33:41 ntpd[7816]: restrict ::: KOD does nothing without LIMITED. 2022-04-09T11:33:41.598 INFO:teuthology.orchestra.run.smithi119.stdout: 9 Apr 11:33:41 ntpd[7816]: Listen and drop on 0 v6wildcard [::]:123 2022-04-09T11:33:41.598 INFO:teuthology.orchestra.run.smithi119.stdout: 9 Apr 11:33:41 ntpd[7816]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-04-09T11:33:41.599 INFO:teuthology.orchestra.run.smithi119.stdout: 9 Apr 11:33:41 ntpd[7816]: Listen normally on 2 lo 127.0.0.1:123 2022-04-09T11:33:41.599 INFO:teuthology.orchestra.run.smithi119.stdout: 9 Apr 11:33:41 ntpd[7816]: Listen normally on 3 enp3s0f1 172.21.15.119:123 2022-04-09T11:33:41.599 INFO:teuthology.orchestra.run.smithi119.stdout: 9 Apr 11:33:41 ntpd[7816]: Listen normally on 4 lo [::1]:123 2022-04-09T11:33:41.599 INFO:teuthology.orchestra.run.smithi119.stdout: 9 Apr 11:33:41 ntpd[7816]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d5ef%5]:123 2022-04-09T11:33:41.600 INFO:teuthology.orchestra.run.smithi119.stdout: 9 Apr 11:33:41 ntpd[7816]: Listening on routing socket on fd #22 for interface updates 2022-04-09T11:33:42.595 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:42 ntpd[7816]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:33:42.596 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:42 ntpd[7816]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:33:42.596 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:42 ntpd[7816]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:33:42.596 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:42 ntpd[7816]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:33:42.596 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:42 ntpd[7816]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-09T11:33:42.597 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:42 ntpd[7816]: can't open /var/log/ntpstats/peerstats.20220409: Permission denied 2022-04-09T11:33:43.596 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:43 ntpd[7816]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:33:43.596 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:43 ntpd[7816]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:33:43.596 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:43 ntpd[7816]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:33:43.597 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:43 ntpd[7816]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:33:43.597 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:43 ntpd[7816]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-09T11:33:43.597 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:43 ntpd[7816]: can't open /var/log/ntpstats/peerstats.20220409: Permission denied 2022-04-09T11:33:44.596 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:44 ntpd[7816]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:33:44.597 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:44 ntpd[7816]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:33:44.597 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:44 ntpd[7816]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:33:44.597 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:44 ntpd[7816]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:33:44.597 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:44 ntpd[7816]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-09T11:33:44.598 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:44 ntpd[7816]: can't open /var/log/ntpstats/peerstats.20220409: Permission denied 2022-04-09T11:33:45.597 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:45 ntpd[7816]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:33:45.598 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:45 ntpd[7816]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:33:45.598 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:45 ntpd[7816]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:33:45.598 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:45 ntpd[7816]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:33:45.599 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:45 ntpd[7816]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:33:45.599 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:45 ntpd[7816]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:33:45.599 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:45 ntpd[7816]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-09T11:33:45.599 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:45 ntpd[7816]: can't open /var/log/ntpstats/peerstats.20220409: Permission denied 2022-04-09T11:33:45.600 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:45 ntpd[7816]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:33:45.600 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:45 ntpd[7816]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:33:45.600 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:45 ntpd[7816]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-09T11:33:45.600 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:45 ntpd[7816]: can't open /var/log/ntpstats/peerstats.20220409: Permission denied 2022-04-09T11:33:46.598 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:46 ntpd[7816]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:33:46.598 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:46 ntpd[7816]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:33:46.598 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:46 ntpd[7816]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:33:46.599 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:46 ntpd[7816]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:33:46.599 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:46 ntpd[7816]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:33:46.599 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:46 ntpd[7816]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:33:46.599 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:46 ntpd[7816]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-09T11:33:46.600 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:46 ntpd[7816]: can't open /var/log/ntpstats/peerstats.20220409: Permission denied 2022-04-09T11:33:46.600 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:46 ntpd[7816]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:33:46.600 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:46 ntpd[7816]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:33:46.600 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:46 ntpd[7816]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-09T11:33:46.601 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:46 ntpd[7816]: can't open /var/log/ntpstats/peerstats.20220409: Permission denied 2022-04-09T11:33:47.598 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:47 ntpd[7816]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:33:47.599 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:47 ntpd[7816]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:33:47.599 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:47 ntpd[7816]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:33:47.599 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:47 ntpd[7816]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:33:47.599 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:47 ntpd[7816]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:33:47.599 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:47 ntpd[7816]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:33:47.600 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:47 ntpd[7816]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-09T11:33:47.600 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:47 ntpd[7816]: can't open /var/log/ntpstats/peerstats.20220409: Permission denied 2022-04-09T11:33:47.600 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:47 ntpd[7816]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:33:47.600 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:47 ntpd[7816]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:33:47.601 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:47 ntpd[7816]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-09T11:33:47.601 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:47 ntpd[7816]: can't open /var/log/ntpstats/peerstats.20220409: Permission denied 2022-04-09T11:33:48.599 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:48 ntpd[7816]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:33:48.599 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:48 ntpd[7816]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:33:48.600 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:48 ntpd[7816]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:33:48.600 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:48 ntpd[7816]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:33:48.600 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:48 ntpd[7816]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:33:48.600 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:48 ntpd[7816]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:33:48.601 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:48 ntpd[7816]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-09T11:33:48.601 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:48 ntpd[7816]: can't open /var/log/ntpstats/peerstats.20220409: Permission denied 2022-04-09T11:33:48.601 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:48 ntpd[7816]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-04-09T11:33:48.601 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:48 ntpd[7816]: can't open /var/log/ntpstats/rawstats.20220409: Permission denied 2022-04-09T11:33:48.601 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:48 ntpd[7816]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-04-09T11:33:48.602 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:48 ntpd[7816]: can't open /var/log/ntpstats/peerstats.20220409: Permission denied 2022-04-09T11:33:48.603 INFO:teuthology.orchestra.run.smithi119.stdout: 9 Apr 11:33:48 ntpd[7816]: ntpd: time slew -0.017365 s 2022-04-09T11:33:48.603 INFO:teuthology.orchestra.run.smithi119.stdout:ntpd: time slew -0.017365s 2022-04-09T11:33:48.603 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:48 ntpd[7816]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2022-04-09T11:33:48.604 INFO:teuthology.orchestra.run.smithi119.stderr: 9 Apr 11:33:48 ntpd[7816]: can't open /var/log/ntpstats/loopstats.20220409: Permission denied 2022-04-09T11:33:48.658 INFO:teuthology.orchestra.run.smithi119.stdout: remote refid st t when poll reach delay offset jitter 2022-04-09T11:33:48.658 INFO:teuthology.orchestra.run.smithi119.stdout:============================================================================== 2022-04-09T11:33:48.658 INFO:teuthology.orchestra.run.smithi119.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-04-09T11:33:48.659 INFO:teuthology.orchestra.run.smithi119.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-04-09T11:33:48.659 INFO:teuthology.orchestra.run.smithi119.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-04-09T11:33:48.659 INFO:teuthology.orchestra.run.smithi119.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-04-09T11:33:48.659 INFO:teuthology.run_tasks:Running task kubeadm... 2022-04-09T11:33:48.674 INFO:tasks.kubeadm:Kubeadm start 2022-04-09T11:33:48.675 INFO:tasks.kubeadm:Config: {'pod_network': 'flannel', 'version': '1.21'} 2022-04-09T11:33:48.675 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-04-09T11:33:48.699 INFO:teuthology.orchestra.run.smithi119.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-04-09T11:33:48.706 INFO:teuthology.orchestra.run.smithi119.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-04-09T11:33:48.713 INFO:teuthology.orchestra.run.smithi119.stdout:net.ipv4.ip_forward = 1 2022-04-09T11:33:48.754 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-04-09T11:33:48.755 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-04-09T11:33:48.770 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-04-09T11:33:48.771 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/docker/daemon.json 2022-04-09T11:33:48.831 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl restart docker || true 2022-04-09T11:33:49.903 INFO:tasks.kubeadm:Identifying pod subnet 2022-04-09T11:33:49.910 INFO:tasks.kubeadm:172.21.15.119 is in 172.21.0.0/20 at pos 3958 2022-04-09T11:33:49.923 INFO:tasks.kubeadm:Pod subnet: 10.251.176.0/21 2022-04-09T11:33:49.923 INFO:tasks.kubeadm:Installing kube{adm,ctl,let} 2022-04-09T11:33:49.924 DEBUG:teuthology.orchestra.run.smithi119:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-04-09T11:33:49.950 INFO:teuthology.orchestra.run.smithi119.stderr: 2022-04-09T11:33:49.950 INFO:teuthology.orchestra.run.smithi119.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-04-09T11:33:49.951 INFO:teuthology.orchestra.run.smithi119.stderr: 2022-04-09T11:33:50.191 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-04-09T11:33:50.192 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-04-09T11:33:50.255 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-04-09T11:33:50.340 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-04-09T11:33:51.689 INFO:teuthology.orchestra.run.smithi119.stdout:Reading package lists... 2022-04-09T11:33:51.863 INFO:teuthology.orchestra.run.smithi119.stdout:Building dependency tree... 2022-04-09T11:33:51.864 INFO:teuthology.orchestra.run.smithi119.stdout:Reading state information... 2022-04-09T11:33:51.955 INFO:teuthology.orchestra.run.smithi119.stdout:8 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-04-09T11:33:51.969 INFO:teuthology.orchestra.run.smithi119.stderr: 2022-04-09T11:33:51.969 INFO:teuthology.orchestra.run.smithi119.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-04-09T11:33:51.969 INFO:teuthology.orchestra.run.smithi119.stderr: 2022-04-09T11:33:52.020 INFO:teuthology.orchestra.run.smithi119.stdout:Reading package lists... 2022-04-09T11:33:52.180 INFO:teuthology.orchestra.run.smithi119.stdout:Building dependency tree... 2022-04-09T11:33:52.180 INFO:teuthology.orchestra.run.smithi119.stdout:Reading state information... 2022-04-09T11:33:52.341 INFO:teuthology.orchestra.run.smithi119.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-04-09T11:33:52.342 INFO:teuthology.orchestra.run.smithi119.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-04-09T11:33:52.342 INFO:teuthology.orchestra.run.smithi119.stdout:The following NEW packages will be installed: 2022-04-09T11:33:52.342 INFO:teuthology.orchestra.run.smithi119.stdout: apt-transport-https 2022-04-09T11:33:52.400 INFO:teuthology.orchestra.run.smithi119.stdout:0 upgraded, 1 newly installed, 0 to remove and 8 not upgraded. 2022-04-09T11:33:52.400 INFO:teuthology.orchestra.run.smithi119.stdout:Need to get 4,680 B of archives. 2022-04-09T11:33:52.400 INFO:teuthology.orchestra.run.smithi119.stdout:After this operation, 162 kB of additional disk space will be used. 2022-04-09T11:33:52.400 INFO:teuthology.orchestra.run.smithi119.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-04-09T11:33:52.611 INFO:teuthology.orchestra.run.smithi119.stderr:debconf: unable to initialize frontend: Dialog 2022-04-09T11:33:52.612 INFO:teuthology.orchestra.run.smithi119.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-04-09T11:33:52.612 INFO:teuthology.orchestra.run.smithi119.stderr:debconf: falling back to frontend: Readline 2022-04-09T11:33:52.633 INFO:teuthology.orchestra.run.smithi119.stderr:debconf: unable to initialize frontend: Readline 2022-04-09T11:33:52.633 INFO:teuthology.orchestra.run.smithi119.stderr:debconf: (This frontend requires a controlling tty.) 2022-04-09T11:33:52.633 INFO:teuthology.orchestra.run.smithi119.stderr:debconf: falling back to frontend: Teletype 2022-04-09T11:33:52.638 INFO:teuthology.orchestra.run.smithi119.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-04-09T11:33:52.692 INFO:teuthology.orchestra.run.smithi119.stdout:Fetched 4,680 B in 0s (91.2 kB/s) 2022-04-09T11:33:52.802 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package apt-transport-https. 2022-04-09T11:33:52.841 INFO:teuthology.orchestra.run.smithi119.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109370 files and directories currently installed.) 2022-04-09T11:33:52.844 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-04-09T11:33:52.885 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-04-09T11:33:53.156 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up apt-transport-https (2.0.6) ... 2022-04-09T11:33:53.716 INFO:teuthology.orchestra.run.smithi119.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-04-09T11:33:53.730 INFO:teuthology.orchestra.run.smithi119.stderr: 2022-04-09T11:33:53.730 INFO:teuthology.orchestra.run.smithi119.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-04-09T11:33:53.730 INFO:teuthology.orchestra.run.smithi119.stderr: 2022-04-09T11:33:53.872 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-04-09T11:33:53.872 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-04-09T11:33:53.888 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-04-09T11:33:53.915 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-04-09T11:33:54.942 INFO:teuthology.orchestra.run.smithi119.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-04-09T11:33:55.056 INFO:teuthology.orchestra.run.smithi119.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [54.7 kB] 2022-04-09T11:33:55.178 INFO:teuthology.orchestra.run.smithi119.stdout:Fetched 64.1 kB in 1s (48.9 kB/s) 2022-04-09T11:33:56.031 INFO:teuthology.orchestra.run.smithi119.stdout:Reading package lists... 2022-04-09T11:33:56.210 INFO:teuthology.orchestra.run.smithi119.stdout:Building dependency tree... 2022-04-09T11:33:56.211 INFO:teuthology.orchestra.run.smithi119.stdout:Reading state information... 2022-04-09T11:33:56.306 INFO:teuthology.orchestra.run.smithi119.stdout:8 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-04-09T11:33:56.320 INFO:teuthology.orchestra.run.smithi119.stderr: 2022-04-09T11:33:56.321 INFO:teuthology.orchestra.run.smithi119.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-04-09T11:33:56.321 INFO:teuthology.orchestra.run.smithi119.stderr: 2022-04-09T11:33:56.374 INFO:teuthology.orchestra.run.smithi119.stdout:Reading package lists... 2022-04-09T11:33:56.552 INFO:teuthology.orchestra.run.smithi119.stdout:Building dependency tree... 2022-04-09T11:33:56.552 INFO:teuthology.orchestra.run.smithi119.stdout:Reading state information... 2022-04-09T11:33:56.690 INFO:teuthology.orchestra.run.smithi119.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-04-09T11:33:56.690 INFO:teuthology.orchestra.run.smithi119.stdout:bridge-utils set to manually installed. 2022-04-09T11:33:56.691 INFO:teuthology.orchestra.run.smithi119.stdout:The following additional packages will be installed: 2022-04-09T11:33:56.691 INFO:teuthology.orchestra.run.smithi119.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-04-09T11:33:56.692 INFO:teuthology.orchestra.run.smithi119.stdout:Suggested packages: 2022-04-09T11:33:56.692 INFO:teuthology.orchestra.run.smithi119.stdout: nftables 2022-04-09T11:33:56.743 INFO:teuthology.orchestra.run.smithi119.stdout:The following NEW packages will be installed: 2022-04-09T11:33:56.744 INFO:teuthology.orchestra.run.smithi119.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-04-09T11:33:56.810 INFO:teuthology.orchestra.run.smithi119.stdout:0 upgraded, 8 newly installed, 0 to remove and 8 not upgraded. 2022-04-09T11:33:56.811 INFO:teuthology.orchestra.run.smithi119.stdout:Need to get 77.7 MB of archives. 2022-04-09T11:33:56.811 INFO:teuthology.orchestra.run.smithi119.stdout:After this operation, 335 MB of additional disk space will be used. 2022-04-09T11:33:56.811 INFO:teuthology.orchestra.run.smithi119.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-04-09T11:33:56.851 INFO:teuthology.orchestra.run.smithi119.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-04-09T11:33:56.878 INFO:teuthology.orchestra.run.smithi119.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-04-09T11:33:56.968 INFO:teuthology.orchestra.run.smithi119.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.23.0-00 [15.3 MB] 2022-04-09T11:33:57.403 INFO:teuthology.orchestra.run.smithi119.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-04-09T11:33:57.977 INFO:teuthology.orchestra.run.smithi119.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.5-00 [19.5 MB] 2022-04-09T11:33:58.455 INFO:teuthology.orchestra.run.smithi119.stdout:Get:7 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.5-00 [8,931 kB] 2022-04-09T11:33:58.594 INFO:teuthology.orchestra.run.smithi119.stdout:Get:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.5-00 [8,582 kB] 2022-04-09T11:33:58.961 INFO:teuthology.orchestra.run.smithi119.stderr:debconf: unable to initialize frontend: Dialog 2022-04-09T11:33:58.961 INFO:teuthology.orchestra.run.smithi119.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-04-09T11:33:58.962 INFO:teuthology.orchestra.run.smithi119.stderr:debconf: falling back to frontend: Readline 2022-04-09T11:33:58.970 INFO:teuthology.orchestra.run.smithi119.stderr:debconf: unable to initialize frontend: Readline 2022-04-09T11:33:58.970 INFO:teuthology.orchestra.run.smithi119.stderr:debconf: (This frontend requires a controlling tty.) 2022-04-09T11:33:58.970 INFO:teuthology.orchestra.run.smithi119.stderr:debconf: falling back to frontend: Teletype 2022-04-09T11:33:58.975 INFO:teuthology.orchestra.run.smithi119.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-04-09T11:33:59.023 INFO:teuthology.orchestra.run.smithi119.stdout:Fetched 77.7 MB in 2s (38.9 MB/s) 2022-04-09T11:33:59.093 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package conntrack. 2022-04-09T11:33:59.134 INFO:teuthology.orchestra.run.smithi119.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109374 files and directories currently installed.) 2022-04-09T11:33:59.137 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-04-09T11:33:59.166 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-04-09T11:33:59.494 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package cri-tools. 2022-04-09T11:33:59.507 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../1-cri-tools_1.23.0-00_amd64.deb ... 2022-04-09T11:33:59.552 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking cri-tools (1.23.0-00) ... 2022-04-09T11:34:00.913 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ebtables. 2022-04-09T11:34:00.926 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-04-09T11:34:00.996 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-04-09T11:34:01.258 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package kubernetes-cni. 2022-04-09T11:34:01.271 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-04-09T11:34:01.316 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-04-09T11:34:02.735 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package socat. 2022-04-09T11:34:02.748 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-04-09T11:34:02.785 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking socat (1.7.3.3-2) ... 2022-04-09T11:34:03.222 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package kubelet. 2022-04-09T11:34:03.235 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../5-kubelet_1.23.5-00_amd64.deb ... 2022-04-09T11:34:03.272 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking kubelet (1.23.5-00) ... 2022-04-09T11:34:05.691 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package kubectl. 2022-04-09T11:34:05.704 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../6-kubectl_1.23.5-00_amd64.deb ... 2022-04-09T11:34:05.741 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking kubectl (1.23.5-00) ... 2022-04-09T11:34:06.952 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package kubeadm. 2022-04-09T11:34:06.965 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../7-kubeadm_1.23.5-00_amd64.deb ... 2022-04-09T11:34:07.010 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking kubeadm (1.23.5-00) ... 2022-04-09T11:34:08.132 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-04-09T11:34:08.272 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up kubectl (1.23.5-00) ... 2022-04-09T11:34:08.398 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-04-09T11:34:09.326 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up socat (1.7.3.3-2) ... 2022-04-09T11:34:09.544 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up cri-tools (1.23.0-00) ... 2022-04-09T11:34:09.879 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-04-09T11:34:10.005 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up kubelet (1.23.5-00) ... 2022-04-09T11:34:10.223 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-04-09T11:34:10.715 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up kubeadm (1.23.5-00) ... 2022-04-09T11:34:11.125 INFO:teuthology.orchestra.run.smithi119.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-04-09T11:34:11.973 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-04-09T11:34:25.866 INFO:teuthology.orchestra.run.smithi119.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.5 2022-04-09T11:34:28.718 INFO:teuthology.orchestra.run.smithi119.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.5 2022-04-09T11:34:31.174 INFO:teuthology.orchestra.run.smithi119.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.5 2022-04-09T11:34:34.614 INFO:teuthology.orchestra.run.smithi119.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.5 2022-04-09T11:34:35.954 INFO:teuthology.orchestra.run.smithi119.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-04-09T11:34:42.447 INFO:teuthology.orchestra.run.smithi119.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-04-09T11:34:44.388 INFO:teuthology.orchestra.run.smithi119.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-04-09T11:34:44.392 INFO:tasks.kubeadm:Token: abcdef.5tckoe9ff24dkmhk 2022-04-09T11:34:44.393 INFO:tasks.kubeadm:Remotes: {Remote(name='ubuntu@smithi119.front.sepia.ceph.com'): '172.21.15.119'} 2022-04-09T11:34:44.393 DEBUG:teuthology.orchestra.run.smithi119:> sudo kubeadm init --node-name smithi119 --token abcdef.5tckoe9ff24dkmhk --pod-network-cidr 10.251.176.0/21 2022-04-09T11:34:44.582 INFO:teuthology.orchestra.run.smithi119.stdout:[init] Using Kubernetes version: v1.23.5 2022-04-09T11:34:44.583 INFO:teuthology.orchestra.run.smithi119.stdout:[preflight] Running pre-flight checks 2022-04-09T11:34:44.859 INFO:teuthology.orchestra.run.smithi119.stdout:[preflight] Pulling images required for setting up a Kubernetes cluster 2022-04-09T11:34:44.859 INFO:teuthology.orchestra.run.smithi119.stdout:[preflight] This might take a minute or two, depending on the speed of your internet connection 2022-04-09T11:34:44.859 INFO:teuthology.orchestra.run.smithi119.stdout:[preflight] You can also perform this action in beforehand using 'kubeadm config images pull' 2022-04-09T11:34:45.158 INFO:teuthology.orchestra.run.smithi119.stdout:[certs] Using certificateDir folder "/etc/kubernetes/pki" 2022-04-09T11:34:45.214 INFO:teuthology.orchestra.run.smithi119.stdout:[certs] Generating "ca" certificate and key 2022-04-09T11:34:45.309 INFO:teuthology.orchestra.run.smithi119.stdout:[certs] Generating "apiserver" certificate and key 2022-04-09T11:34:45.310 INFO:teuthology.orchestra.run.smithi119.stdout:[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local smithi119] and IPs [10.96.0.1 172.21.15.119] 2022-04-09T11:34:45.434 INFO:teuthology.orchestra.run.smithi119.stdout:[certs] Generating "apiserver-kubelet-client" certificate and key 2022-04-09T11:34:45.635 INFO:teuthology.orchestra.run.smithi119.stdout:[certs] Generating "front-proxy-ca" certificate and key 2022-04-09T11:34:45.712 INFO:teuthology.orchestra.run.smithi119.stdout:[certs] Generating "front-proxy-client" certificate and key 2022-04-09T11:34:45.848 INFO:teuthology.orchestra.run.smithi119.stdout:[certs] Generating "etcd/ca" certificate and key 2022-04-09T11:34:45.906 INFO:teuthology.orchestra.run.smithi119.stdout:[certs] Generating "etcd/server" certificate and key 2022-04-09T11:34:45.906 INFO:teuthology.orchestra.run.smithi119.stdout:[certs] etcd/server serving cert is signed for DNS names [localhost smithi119] and IPs [172.21.15.119 127.0.0.1 ::1] 2022-04-09T11:34:46.261 INFO:teuthology.orchestra.run.smithi119.stdout:[certs] Generating "etcd/peer" certificate and key 2022-04-09T11:34:46.261 INFO:teuthology.orchestra.run.smithi119.stdout:[certs] etcd/peer serving cert is signed for DNS names [localhost smithi119] and IPs [172.21.15.119 127.0.0.1 ::1] 2022-04-09T11:34:46.379 INFO:teuthology.orchestra.run.smithi119.stdout:[certs] Generating "etcd/healthcheck-client" certificate and key 2022-04-09T11:34:46.475 INFO:teuthology.orchestra.run.smithi119.stdout:[certs] Generating "apiserver-etcd-client" certificate and key 2022-04-09T11:34:46.749 INFO:teuthology.orchestra.run.smithi119.stdout:[certs] Generating "sa" key and public key 2022-04-09T11:34:46.749 INFO:teuthology.orchestra.run.smithi119.stdout:[kubeconfig] Using kubeconfig folder "/etc/kubernetes" 2022-04-09T11:34:46.891 INFO:teuthology.orchestra.run.smithi119.stdout:[kubeconfig] Writing "admin.conf" kubeconfig file 2022-04-09T11:34:46.982 INFO:teuthology.orchestra.run.smithi119.stdout:[kubeconfig] Writing "kubelet.conf" kubeconfig file 2022-04-09T11:34:47.051 INFO:teuthology.orchestra.run.smithi119.stdout:[kubeconfig] Writing "controller-manager.conf" kubeconfig file 2022-04-09T11:34:47.361 INFO:teuthology.orchestra.run.smithi119.stdout:[kubeconfig] Writing "scheduler.conf" kubeconfig file 2022-04-09T11:34:47.374 INFO:teuthology.orchestra.run.smithi119.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-04-09T11:34:47.376 INFO:teuthology.orchestra.run.smithi119.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-04-09T11:34:47.376 INFO:teuthology.orchestra.run.smithi119.stdout:[kubelet-start] Starting the kubelet 2022-04-09T11:34:47.602 INFO:teuthology.orchestra.run.smithi119.stdout:[control-plane] Using manifest folder "/etc/kubernetes/manifests" 2022-04-09T11:34:47.603 INFO:teuthology.orchestra.run.smithi119.stdout:[control-plane] Creating static Pod manifest for "kube-apiserver" 2022-04-09T11:34:47.605 INFO:teuthology.orchestra.run.smithi119.stdout:[control-plane] Creating static Pod manifest for "kube-controller-manager" 2022-04-09T11:34:47.606 INFO:teuthology.orchestra.run.smithi119.stdout:[control-plane] Creating static Pod manifest for "kube-scheduler" 2022-04-09T11:34:47.607 INFO:teuthology.orchestra.run.smithi119.stdout:[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" 2022-04-09T11:34:47.609 INFO:teuthology.orchestra.run.smithi119.stdout:[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s 2022-04-09T11:35:06.112 INFO:teuthology.orchestra.run.smithi119.stdout:[apiclient] All control plane components are healthy after 18.503227 seconds 2022-04-09T11:35:06.113 INFO:teuthology.orchestra.run.smithi119.stdout:[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace 2022-04-09T11:35:06.155 INFO:teuthology.orchestra.run.smithi119.stdout:[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster 2022-04-09T11:35:06.155 INFO:teuthology.orchestra.run.smithi119.stdout:NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently. 2022-04-09T11:35:06.700 INFO:teuthology.orchestra.run.smithi119.stdout:[upload-certs] Skipping phase. Please see --upload-certs 2022-04-09T11:35:06.700 INFO:teuthology.orchestra.run.smithi119.stdout:[mark-control-plane] Marking the node smithi119 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] 2022-04-09T11:35:06.700 INFO:teuthology.orchestra.run.smithi119.stdout:[mark-control-plane] Marking the node smithi119 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] 2022-04-09T11:35:07.239 INFO:teuthology.orchestra.run.smithi119.stdout:[bootstrap-token] Using token: abcdef.5tckoe9ff24dkmhk 2022-04-09T11:35:07.239 INFO:teuthology.orchestra.run.smithi119.stdout:[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles 2022-04-09T11:35:07.299 INFO:teuthology.orchestra.run.smithi119.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes 2022-04-09T11:35:07.349 INFO:teuthology.orchestra.run.smithi119.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials 2022-04-09T11:35:07.355 INFO:teuthology.orchestra.run.smithi119.stdout:[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token 2022-04-09T11:35:07.445 INFO:teuthology.orchestra.run.smithi119.stdout:[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster 2022-04-09T11:35:07.448 INFO:teuthology.orchestra.run.smithi119.stdout:[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace 2022-04-09T11:35:07.488 INFO:teuthology.orchestra.run.smithi119.stdout:[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key 2022-04-09T11:35:07.973 INFO:teuthology.orchestra.run.smithi119.stdout:[addons] Applied essential addon: CoreDNS 2022-04-09T11:35:08.051 INFO:teuthology.orchestra.run.smithi119.stdout:[addons] Applied essential addon: kube-proxy 2022-04-09T11:35:08.052 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:35:08.053 INFO:teuthology.orchestra.run.smithi119.stdout:Your Kubernetes control-plane has initialized successfully! 2022-04-09T11:35:08.053 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:35:08.053 INFO:teuthology.orchestra.run.smithi119.stdout:To start using your cluster, you need to run the following as a regular user: 2022-04-09T11:35:08.053 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:35:08.053 INFO:teuthology.orchestra.run.smithi119.stdout: mkdir -p $HOME/.kube 2022-04-09T11:35:08.054 INFO:teuthology.orchestra.run.smithi119.stdout: sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 2022-04-09T11:35:08.054 INFO:teuthology.orchestra.run.smithi119.stdout: sudo chown $(id -u):$(id -g) $HOME/.kube/config 2022-04-09T11:35:08.054 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:35:08.054 INFO:teuthology.orchestra.run.smithi119.stdout:Alternatively, if you are the root user, you can run: 2022-04-09T11:35:08.055 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:35:08.055 INFO:teuthology.orchestra.run.smithi119.stdout: export KUBECONFIG=/etc/kubernetes/admin.conf 2022-04-09T11:35:08.055 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:35:08.055 INFO:teuthology.orchestra.run.smithi119.stdout:You should now deploy a pod network to the cluster. 2022-04-09T11:35:08.055 INFO:teuthology.orchestra.run.smithi119.stdout:Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: 2022-04-09T11:35:08.056 INFO:teuthology.orchestra.run.smithi119.stdout: https://kubernetes.io/docs/concepts/cluster-administration/addons/ 2022-04-09T11:35:08.056 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:35:08.056 INFO:teuthology.orchestra.run.smithi119.stdout:Then you can join any number of worker nodes by running the following on each as root: 2022-04-09T11:35:08.056 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:35:08.057 INFO:teuthology.orchestra.run.smithi119.stdout:kubeadm join 172.21.15.119:6443 --token abcdef.5tckoe9ff24dkmhk \ 2022-04-09T11:35:08.057 INFO:teuthology.orchestra.run.smithi119.stdout: --discovery-token-ca-cert-hash sha256:0ff3a390815a806729d06041cc2b3bc804fb4233f9dc66ee53e8fdcc7c3ed073 2022-04-09T11:35:08.058 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-04-09T11:35:08.059 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/etc/kubernetes/admin.conf of=/dev/stdout 2022-04-09T11:35:08.070 INFO:tasks.kubeadm:Setting up kubectl 2022-04-09T11:35:08.070 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-04-09T11:35:08.136 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-04-09T11:35:08.136 DEBUG:teuthology.orchestra.run.smithi119:> dd of=.kube/config 2022-04-09T11:35:08.142 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-04-09T11:35:08.142 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/root/.kube/config 2022-04-09T11:35:08.194 DEBUG:teuthology.orchestra.run.smithi119:> curl https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml 2022-04-09T11:35:08.207 INFO:teuthology.orchestra.run.smithi119.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-04-09T11:35:08.207 INFO:teuthology.orchestra.run.smithi119.stderr: Dload Upload Total Spent Left Speed 2022-04-09T11:35:08.258 INFO:teuthology.orchestra.run.smithi119.stdout:--- 2022-04-09T11:35:08.258 INFO:teuthology.orchestra.run.smithi119.stdout:apiVersion: policy/v1beta1 2022-04-09T11:35:08.258 INFO:teuthology.orchestra.run.smithi119.stdout:kind: PodSecurityPolicy 2022-04-09T11:35:08.259 INFO:teuthology.orchestra.run.smithi119.stdout:metadata: 2022-04-09T11:35:08.259 INFO:teuthology.orchestra.run.smithi119.stdout: name: psp.flannel.unprivileged 2022-04-09T11:35:08.259 INFO:teuthology.orchestra.run.smithi119.stdout: annotations: 2022-04-09T11:35:08.259 INFO:teuthology.orchestra.run.smithi119.stdout: seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default 2022-04-09T11:35:08.259 INFO:teuthology.orchestra.run.smithi119.stdout: seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default 2022-04-09T11:35:08.260 INFO:teuthology.orchestra.run.smithi119.stdout: apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default 2022-04-09T11:35:08.260 INFO:teuthology.orchestra.run.smithi119.stdout: apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default 2022-04-09T11:35:08.260 INFO:teuthology.orchestra.run.smithi119.stdout:spec: 2022-04-09T11:35:08.260 INFO:teuthology.orchestra.run.smithi119.stdout: privileged: false 2022-04-09T11:35:08.261 INFO:teuthology.orchestra.run.smithi119.stdout: volumes: 2022-04-09T11:35:08.261 INFO:teuthology.orchestra.run.smithi119.stdout: - configMap 2022-04-09T11:35:08.261 INFO:teuthology.orchestra.run.smithi119.stdout: - secret 2022-04-09T11:35:08.261 INFO:teuthology.orchestra.run.smithi119.stdout: - emptyDir 2022-04-09T11:35:08.261 INFO:teuthology.orchestra.run.smithi119.stdout: - hostPath 2022-04-09T11:35:08.262 INFO:teuthology.orchestra.run.smithi119.stdout: allowedHostPaths: 2022-04-09T11:35:08.262 INFO:teuthology.orchestra.run.smithi119.stdout: - pathPrefix: "/etc/cni/net.d" 2022-04-09T11:35:08.262 INFO:teuthology.orchestra.run.smithi119.stdout: - pathPrefix: "/etc/kube-flannel" 2022-04-09T11:35:08.262 INFO:teuthology.orchestra.run.smithi119.stdout: - pathPrefix: "/run/flannel" 2022-04-09T11:35:08.263 INFO:teuthology.orchestra.run.smithi119.stdout: readOnlyRootFilesystem: false 2022-04-09T11:35:08.263 INFO:teuthology.orchestra.run.smithi119.stdout: # Users and groups 2022-04-09T11:35:08.263 INFO:teuthology.orchestra.run.smithi119.stdout: runAsUser: 2022-04-09T11:35:08.263 INFO:teuthology.orchestra.run.smithi119.stdout: rule: RunAsAny 2022-04-09T11:35:08.263 INFO:teuthology.orchestra.run.smithi119.stdout: supplementalGroups: 2022-04-09T11:35:08.264 INFO:teuthology.orchestra.run.smithi119.stdout: rule: RunAsAny 2022-04-09T11:35:08.264 INFO:teuthology.orchestra.run.smithi119.stdout: fsGroup: 2022-04-09T11:35:08.264 INFO:teuthology.orchestra.run.smithi119.stdout: rule: RunAsAny 2022-04-09T11:35:08.264 INFO:teuthology.orchestra.run.smithi119.stdout: # Privilege Escalation 2022-04-09T11:35:08.264 INFO:teuthology.orchestra.run.smithi119.stdout: allowPrivilegeEscalation: false 2022-04-09T11:35:08.265 INFO:teuthology.orchestra.run.smithi119.stdout: defaultAllowPrivilegeEscalation: false 2022-04-09T11:35:08.265 INFO:teuthology.orchestra.run.smithi119.stdout: # Capabilities 2022-04-09T11:35:08.265 INFO:teuthology.orchestra.run.smithi119.stdout: allowedCapabilities: ['NET_ADMIN', 'NET_RAW'] 2022-04-09T11:35:08.265 INFO:teuthology.orchestra.run.smithi119.stdout: defaultAddCapabilities: [] 2022-04-09T11:35:08.266 INFO:teuthology.orchestra.run.smithi119.stdout: requiredDropCapabilities: [] 2022-04-09T11:35:08.266 INFO:teuthology.orchestra.run.smithi119.stdout: # Host namespaces 2022-04-09T11:35:08.266 INFO:teuthology.orchestra.run.smithi119.stdout: hostPID: false 2022-04-09T11:35:08.266 INFO:teuthology.orchestra.run.smithi119.stdout: hostIPC: false 2022-04-09T11:35:08.266 INFO:teuthology.orchestra.run.smithi119.stdout: hostNetwork: true 2022-04-09T11:35:08.267 INFO:teuthology.orchestra.run.smithi119.stdout: hostPorts: 2022-04-09T11:35:08.267 INFO:teuthology.orchestra.run.smithi119.stdout: - min: 0 2022-04-09T11:35:08.267 INFO:teuthology.orchestra.run.smithi119.stdout: max: 65535 2022-04-09T11:35:08.267 INFO:teuthology.orchestra.run.smithi119.stdout: # SELinux 2022-04-09T11:35:08.268 INFO:teuthology.orchestra.run.smithi119.stdout: seLinux: 2022-04-09T11:35:08.268 INFO:teuthology.orchestra.run.smithi119.stdout: # SELinux is unused in CaaSP 2022-04-09T11:35:08.268 INFO:teuthology.orchestra.run.smithi119.stdout: rule: 'RunAsAny' 2022-04-09T11:35:08.268 INFO:teuthology.orchestra.run.smithi119.stdout:--- 2022-04-09T11:35:08.268 INFO:teuthology.orchestra.run.smithi119.stdout:kind: ClusterRole 2022-04-09T11:35:08.269 INFO:teuthology.orchestra.run.smithi119.stdout:apiVersion: rbac.authorization.k8s.io/v1 2022-04-09T11:35:08.269 INFO:teuthology.orchestra.run.smithi119.stdout:metadata: 2022-04-09T11:35:08.269 INFO:teuthology.orchestra.run.smithi119.stdout: name: flannel 2022-04-09T11:35:08.269 INFO:teuthology.orchestra.run.smithi119.stdout:rules: 2022-04-09T11:35:08.269 INFO:teuthology.orchestra.run.smithi119.stdout:- apiGroups: ['extensions'] 2022-04-09T11:35:08.270 INFO:teuthology.orchestra.run.smithi119.stdout: resources: ['podsecuritypolicies'] 2022-04-09T11:35:08.270 INFO:teuthology.orchestra.run.smithi119.stdout: verbs: ['use'] 2022-04-09T11:35:08.270 INFO:teuthology.orchestra.run.smithi119.stdout: resourceNames: ['psp.flannel.unprivileged'] 2022-04-09T11:35:08.270 INFO:teuthology.orchestra.run.smithi119.stdout:- apiGroups: 2022-04-09T11:35:08.271 INFO:teuthology.orchestra.run.smithi119.stdout: - "" 2022-04-09T11:35:08.271 INFO:teuthology.orchestra.run.smithi119.stdout: resources: 2022-04-09T11:35:08.271 INFO:teuthology.orchestra.run.smithi119.stdout: - pods 2022-04-09T11:35:08.271 INFO:teuthology.orchestra.run.smithi119.stdout: verbs: 2022-04-09T11:35:08.271 INFO:teuthology.orchestra.run.smithi119.stdout: - get 2022-04-09T11:35:08.272 INFO:teuthology.orchestra.run.smithi119.stdout:- apiGroups: 2022-04-09T11:35:08.272 INFO:teuthology.orchestra.run.smithi119.stdout: - "" 2022-04-09T11:35:08.272 INFO:teuthology.orchestra.run.smithi119.stdout: resources: 2022-04-09T11:35:08.272 INFO:teuthology.orchestra.run.smithi119.stdout: - nodes 2022-04-09T11:35:08.272 INFO:teuthology.orchestra.run.smithi119.stdout: verbs: 2022-04-09T11:35:08.273 INFO:teuthology.orchestra.run.smithi119.stdout: - list 2022-04-09T11:35:08.273 INFO:teuthology.orchestra.run.smithi119.stdout: - watch 2022-04-09T11:35:08.273 INFO:teuthology.orchestra.run.smithi119.stdout:- apiGroups: 2022-04-09T11:35:08.273 INFO:teuthology.orchestra.run.smithi119.stdout: - "" 2022-04-09T11:35:08.274 INFO:teuthology.orchestra.run.smithi119.stdout: resources: 2022-04-09T11:35:08.274 INFO:teuthology.orchestra.run.smithi119.stdout: - nodes/status 2022-04-09T11:35:08.274 INFO:teuthology.orchestra.run.smithi119.stdout: verbs: 2022-04-09T11:35:08.274 INFO:teuthology.orchestra.run.smithi119.stdout: - patch 2022-04-09T11:35:08.274 INFO:teuthology.orchestra.run.smithi119.stdout:--- 2022-04-09T11:35:08.275 INFO:teuthology.orchestra.run.smithi119.stdout:kind: ClusterRoleBinding 2022-04-09T11:35:08.275 INFO:teuthology.orchestra.run.smithi119.stdout:apiVersion: rbac.authorization.k8s.io/v1 2022-04-09T11:35:08.275 INFO:teuthology.orchestra.run.smithi119.stdout:metadata: 2022-04-09T11:35:08.275 INFO:teuthology.orchestra.run.smithi119.stdout: name: flannel 2022-04-09T11:35:08.276 INFO:teuthology.orchestra.run.smithi119.stdout:roleRef: 2022-04-09T11:35:08.276 INFO:teuthology.orchestra.run.smithi119.stdout: apiGroup: rbac.authorization.k8s.io 2022-04-09T11:35:08.276 INFO:teuthology.orchestra.run.smithi119.stdout: kind: ClusterRole 2022-04-09T11:35:08.276 INFO:teuthology.orchestra.run.smithi119.stdout: name: flannel 2022-04-09T11:35:08.276 INFO:teuthology.orchestra.run.smithi119.stdout:subjects: 2022-04-09T11:35:08.277 INFO:teuthology.orchestra.run.smithi119.stdout:- kind: ServiceAccount 2022-04-09T11:35:08.277 INFO:teuthology.orchestra.run.smithi119.stdout: name: flannel 2022-04-09T11:35:08.277 INFO:teuthology.orchestra.run.smithi119.stdout: namespace: kube-system 2022-04-09T11:35:08.277 INFO:teuthology.orchestra.run.smithi119.stdout:--- 2022-04-09T11:35:08.277 INFO:teuthology.orchestra.run.smithi119.stdout:apiVersion: v1 2022-04-09T11:35:08.278 INFO:teuthology.orchestra.run.smithi119.stdout:kind: ServiceAccount 2022-04-09T11:35:08.278 INFO:teuthology.orchestra.run.smithi119.stdout:metadata: 2022-04-09T11:35:08.278 INFO:teuthology.orchestra.run.smithi119.stdout: name: flannel 2022-04-09T11:35:08.278 INFO:teuthology.orchestra.run.smithi119.stdout: namespace: kube-system 2022-04-09T11:35:08.279 INFO:teuthology.orchestra.run.smithi119.stdout:--- 2022-04-09T11:35:08.279 INFO:teuthology.orchestra.run.smithi119.stdout:kind: ConfigMap 2022-04-09T11:35:08.279 INFO:teuthology.orchestra.run.smithi119.stdout:apiVersion: v1 2022-04-09T11:35:08.279 INFO:teuthology.orchestra.run.smithi119.stdout:metadata: 2022-04-09T11:35:08.279 INFO:teuthology.orchestra.run.smithi119.stdout: name: kube-flannel-cfg 2022-04-09T11:35:08.280 INFO:teuthology.orchestra.run.smithi119.stdout: namespace: kube-system 2022-04-09T11:35:08.280 INFO:teuthology.orchestra.run.smithi119.stdout: labels: 2022-04-09T11:35:08.280 INFO:teuthology.orchestra.run.smithi119.stdout: tier: node 2022-04-09T11:35:08.280 INFO:teuthology.orchestra.run.smithi119.stdout: app: flannel 2022-04-09T11:35:08.281 INFO:teuthology.orchestra.run.smithi119.stdout:data: 2022-04-09T11:35:08.281 INFO:teuthology.orchestra.run.smithi119.stdout: cni-conf.json: | 2022-04-09T11:35:08.281 INFO:teuthology.orchestra.run.smithi119.stdout: { 2022-04-09T11:35:08.281 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "cbr0", 2022-04-09T11:35:08.281 INFO:teuthology.orchestra.run.smithi119.stdout: "cniVersion": "0.3.1", 2022-04-09T11:35:08.282 INFO:teuthology.orchestra.run.smithi119.stdout: "plugins": [ 2022-04-09T11:35:08.282 INFO:teuthology.orchestra.run.smithi119.stdout: { 2022-04-09T11:35:08.282 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "flannel", 2022-04-09T11:35:08.282 INFO:teuthology.orchestra.run.smithi119.stdout: "delegate": { 2022-04-09T11:35:08.283 INFO:teuthology.orchestra.run.smithi119.stdout: "hairpinMode": true, 2022-04-09T11:35:08.283 INFO:teuthology.orchestra.run.smithi119.stdout: "isDefaultGateway": true 2022-04-09T11:35:08.283 INFO:teuthology.orchestra.run.smithi119.stdout: } 2022-04-09T11:35:08.283 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2022-04-09T11:35:08.283 INFO:teuthology.orchestra.run.smithi119.stdout: { 2022-04-09T11:35:08.284 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "portmap", 2022-04-09T11:35:08.284 INFO:teuthology.orchestra.run.smithi119.stdout: "capabilities": { 2022-04-09T11:35:08.284 INFO:teuthology.orchestra.run.smithi119.stdout: "portMappings": true 2022-04-09T11:35:08.284 INFO:teuthology.orchestra.run.smithi119.stdout: } 2022-04-09T11:35:08.285 INFO:teuthology.orchestra.run.smithi119.stdout: } 2022-04-09T11:35:08.285 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2022-04-09T11:35:08.285 INFO:teuthology.orchestra.run.smithi119.stdout: } 2022-04-09T11:35:08.285 INFO:teuthology.orchestra.run.smithi119.stdout: net-conf.json: | 2022-04-09T11:35:08.285 INFO:teuthology.orchestra.run.smithi119.stdout: { 2022-04-09T11:35:08.286 INFO:teuthology.orchestra.run.smithi119.stdout: "Network": "10.244.0.0/16", 2022-04-09T11:35:08.286 INFO:teuthology.orchestra.run.smithi119.stdout: "Backend": { 2022-04-09T11:35:08.286 INFO:teuthology.orchestra.run.smithi119.stdout: "Type": "vxlan" 2022-04-09T11:35:08.286 INFO:teuthology.orchestra.run.smithi119.stdout: } 2022-04-09T11:35:08.287 INFO:teuthology.orchestra.run.smithi119.stdout: } 2022-04-09T11:35:08.287 INFO:teuthology.orchestra.run.smithi119.stdout:--- 2022-04-09T11:35:08.287 INFO:teuthology.orchestra.run.smithi119.stdout:apiVersion: apps/v1 2022-04-09T11:35:08.287 INFO:teuthology.orchestra.run.smithi119.stdout:kind: DaemonSet 2022-04-09T11:35:08.287 INFO:teuthology.orchestra.run.smithi119.stdout:metadata: 2022-04-09T11:35:08.288 INFO:teuthology.orchestra.run.smithi119.stdout: name: kube-flannel-ds 2022-04-09T11:35:08.288 INFO:teuthology.orchestra.run.smithi119.stdout: namespace: kube-system 2022-04-09T11:35:08.288 INFO:teuthology.orchestra.run.smithi119.stdout: labels: 2022-04-09T11:35:08.288 INFO:teuthology.orchestra.run.smithi119.stdout: tier: node 2022-04-09T11:35:08.288 INFO:teuthology.orchestra.run.smithi119.stdout: app: flannel 2022-04-09T11:35:08.289 INFO:teuthology.orchestra.run.smithi119.stdout:spec: 2022-04-09T11:35:08.289 INFO:teuthology.orchestra.run.smithi119.stdout: selector: 2022-04-09T11:35:08.289 INFO:teuthology.orchestra.run.smithi119.stdout: matchLabels: 2022-04-09T11:35:08.289 INFO:teuthology.orchestra.run.smithi119.stdout: app: flannel 2022-04-09T11:35:08.290 INFO:teuthology.orchestra.run.smithi119.stdout: template: 2022-04-09T11:35:08.290 INFO:teuthology.orchestra.run.smithi119.stdout: metadata: 2022-04-09T11:35:08.290 INFO:teuthology.orchestra.run.smithi119.stdout: labels: 2022-04-09T11:35:08.290 INFO:teuthology.orchestra.run.smithi119.stdout: tier: node 2022-04-09T11:35:08.290 INFO:teuthology.orchestra.run.smithi119.stdout: app: flannel 2022-04-09T11:35:08.291 INFO:teuthology.orchestra.run.smithi119.stdout: spec: 2022-04-09T11:35:08.291 INFO:teuthology.orchestra.run.smithi119.stdout: affinity: 2022-04-09T11:35:08.291 INFO:teuthology.orchestra.run.smithi119.stdout: nodeAffinity: 2022-04-09T11:35:08.291 INFO:teuthology.orchestra.run.smithi119.stdout: requiredDuringSchedulingIgnoredDuringExecution: 2022-04-09T11:35:08.292 INFO:teuthology.orchestra.run.smithi119.stdout: nodeSelectorTerms: 2022-04-09T11:35:08.292 INFO:teuthology.orchestra.run.smithi119.stdout: - matchExpressions: 2022-04-09T11:35:08.292 INFO:teuthology.orchestra.run.smithi119.stdout: - key: kubernetes.io/os 2022-04-09T11:35:08.292 INFO:teuthology.orchestra.run.smithi119.stdout: operator: In 2022-04-09T11:35:08.292 INFO:teuthology.orchestra.run.smithi119.stdout: values: 2022-04-09T11:35:08.293 INFO:teuthology.orchestra.run.smithi119.stdout: - linux 2022-04-09T11:35:08.293 INFO:teuthology.orchestra.run.smithi119.stdout: hostNetwork: true 2022-04-09T11:35:08.293 INFO:teuthology.orchestra.run.smithi119.stdout: priorityClassName: system-node-critical 2022-04-09T11:35:08.293 INFO:teuthology.orchestra.run.smithi119.stdout: tolerations: 2022-04-09T11:35:08.294 INFO:teuthology.orchestra.run.smithi119.stdout: - operator: Exists 2022-04-09T11:35:08.294 INFO:teuthology.orchestra.run.smithi119.stdout: effect: NoSchedule 2022-04-09T11:35:08.294 INFO:teuthology.orchestra.run.smithi119.stdout: serviceAccountName: flannel 2022-04-09T11:35:08.294 INFO:teuthology.orchestra.run.smithi119.stdout: initContainers: 2022-04-09T11:35:08.294 INFO:teuthology.orchestra.run.smithi119.stdout: - name: install-cni-plugin 2022-04-09T11:35:08.295 INFO:teuthology.orchestra.run.smithi119.stdout: #image: flannelcni/flannel-cni-plugin:v1.0.1 for ppc64le and mips64le (dockerhub limitations may apply) 2022-04-09T11:35:08.295 INFO:teuthology.orchestra.run.smithi119.stdout: image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.1 2022-04-09T11:35:08.295 INFO:teuthology.orchestra.run.smithi119.stdout: command: 2022-04-09T11:35:08.295 INFO:teuthology.orchestra.run.smithi119.stdout: - cp 2022-04-09T11:35:08.296 INFO:teuthology.orchestra.run.smithi119.stdout: args: 2022-04-09T11:35:08.296 INFO:teuthology.orchestra.run.smithi119.stdout: - -f 2022-04-09T11:35:08.296 INFO:teuthology.orchestra.run.smithi119.stdout: - /flannel 2022-04-09T11:35:08.296 INFO:teuthology.orchestra.run.smithi119.stdout: - /opt/cni/bin/flannel 2022-04-09T11:35:08.296 INFO:teuthology.orchestra.run.smithi119.stdout: volumeMounts: 2022-04-09T11:35:08.297 INFO:teuthology.orchestra.run.smithi119.stdout: - name: cni-plugin 2022-04-09T11:35:08.297 INFO:teuthology.orchestra.run.smithi119.stdout: mountPath: /opt/cni/bin 2022-04-09T11:35:08.297 INFO:teuthology.orchestra.run.smithi119.stdout: - name: install-cni 2022-04-09T11:35:08.297 INFO:teuthology.orchestra.run.smithi119.stdout: #image: flannelcni/flannel:v0.17.0 for ppc64le and mips64le (dockerhub limitations may apply) 2022-04-09T11:35:08.297 INFO:teuthology.orchestra.run.smithi119.stdout: image: rancher/mirrored-flannelcni-flannel:v0.17.0 2022-04-09T11:35:08.298 INFO:teuthology.orchestra.run.smithi119.stdout: command: 2022-04-09T11:35:08.298 INFO:teuthology.orchestra.run.smithi119.stdout: - cp 2022-04-09T11:35:08.298 INFO:teuthology.orchestra.run.smithi119.stdout: args: 2022-04-09T11:35:08.298 INFO:teuthology.orchestra.run.smithi119.stdout: - -f 2022-04-09T11:35:08.299 INFO:teuthology.orchestra.run.smithi119.stdout: - /etc/kube-flannel/cni-conf.json 2022-04-09T11:35:08.299 INFO:teuthology.orchestra.run.smithi119.stdout: - /etc/cni/net.d/10-flannel.conflist 2022-04-09T11:35:08.299 INFO:teuthology.orchestra.run.smithi119.stdout: volumeMounts: 2022-04-09T11:35:08.299 INFO:teuthology.orchestra.run.smithi119.stdout: - name: cni 2022-04-09T11:35:08.299 INFO:teuthology.orchestra.run.smithi119.stdout: mountPath: /etc/cni/net.d 2022-04-09T11:35:08.300 INFO:teuthology.orchestra.run.smithi119.stdout: - name: flannel-cfg 2022-04-09T11:35:08.300 INFO:teuthology.orchestra.run.smithi119.stdout: mountPath: /etc/kube-flannel/ 2022-04-09T11:35:08.300 INFO:teuthology.orchestra.run.smithi119.stdout: containers: 2022-04-09T11:35:08.300 INFO:teuthology.orchestra.run.smithi119.stdout: - name: kube-flannel 2022-04-09T11:35:08.301 INFO:teuthology.orchestra.run.smithi119.stdout: #image: flannelcni/flannel:v0.17.0 for ppc64le and mips64le (dockerhub limitations may apply) 2022-04-09T11:35:08.301 INFO:teuthology.orchestra.run.smithi119.stdout: image: rancher/mirrored-flannelcni-flannel:v0.17.0 2022-04-09T11:35:08.301 INFO:teuthology.orchestra.run.smithi119.stdout: command: 2022-04-09T11:35:08.301 INFO:teuthology.orchestra.run.smithi119.stdout: - /opt/bin/flanneld 2022-04-09T11:35:08.301 INFO:teuthology.orchestra.run.smithi119.stdout: args: 2022-04-09T11:35:08.302 INFO:teuthology.orchestra.run.smithi119.stdout: - --ip-masq 2022-04-09T11:35:08.302 INFO:teuthology.orchestra.run.smithi119.stdout: - --kube-subnet-mgr 2022-04-09T11:35:08.302 INFO:teuthology.orchestra.run.smithi119.stdout: resources: 2022-04-09T11:35:08.302 INFO:teuthology.orchestra.run.smithi119.stdout: requests: 2022-04-09T11:35:08.302 INFO:teuthology.orchestra.run.smithi119.stdout: cpu: "100m" 2022-04-09T11:35:08.303 INFO:teuthology.orchestra.run.smithi119.stdout: memory: "50Mi" 2022-04-09T11:35:08.303 INFO:teuthology.orchestra.run.smithi119.stdout: limits: 2022-04-09T11:35:08.303 INFO:teuthology.orchestra.run.smithi119.stdout: cpu: "100m" 2022-04-09T11:35:08.303 INFO:teuthology.orchestra.run.smithi119.stdout: memory: "50Mi" 2022-04-09T11:35:08.304 INFO:teuthology.orchestra.run.smithi119.stdout: securityContext: 2022-04-09T11:35:08.304 INFO:teuthology.orchestra.run.smithi119.stdout: privileged: false 2022-04-09T11:35:08.304 INFO:teuthology.orchestra.run.smithi119.stdout: capabilities: 2022-04-09T11:35:08.304 INFO:teuthology.orchestra.run.smithi119.stdout: add: ["NET_ADMIN", "NET_RAW"] 2022-04-09T11:35:08.304 INFO:teuthology.orchestra.run.smithi119.stdout: env: 2022-04-09T11:35:08.305 INFO:teuthology.orchestra.run.smithi119.stdout: - name: POD_NAME 2022-04-09T11:35:08.305 INFO:teuthology.orchestra.run.smithi119.stdout: valueFrom: 2022-04-09T11:35:08.305 INFO:teuthology.orchestra.run.smithi119.stdout: fieldRef: 2022-04-09T11:35:08.305 INFO:teuthology.orchestra.run.smithi119.stdout: fieldPath: metadata.name 2022-04-09T11:35:08.305 INFO:teuthology.orchestra.run.smithi119.stdout: - name: POD_NAMESPACE 2022-04-09T11:35:08.306 INFO:teuthology.orchestra.run.smithi119.stdout: valueFrom: 2022-04-09T11:35:08.306 INFO:teuthology.orchestra.run.smithi119.stdout: fieldRef: 2022-04-09T11:35:08.306 INFO:teuthology.orchestra.run.smithi119.stdout: fieldPath: metadata.namespace 2022-04-09T11:35:08.306 INFO:teuthology.orchestra.run.smithi119.stdout: volumeMounts: 2022-04-09T11:35:08.307 INFO:teuthology.orchestra.run.smithi119.stdout: - name: run 2022-04-09T11:35:08.307 INFO:teuthology.orchestra.run.smithi119.stdout: mountPath: /run/flannel 2022-04-09T11:35:08.307 INFO:teuthology.orchestra.run.smithi119.stdout: - name: flannel-cfg 2022-04-09T11:35:08.307 INFO:teuthology.orchestra.run.smithi119.stdout: mountPath: /etc/kube-flannel/ 2022-04-09T11:35:08.307 INFO:teuthology.orchestra.run.smithi119.stdout: - name: xtables-lock 2022-04-09T11:35:08.308 INFO:teuthology.orchestra.run.smithi119.stdout: mountPath: /run/xtables.lock 2022-04-09T11:35:08.308 INFO:teuthology.orchestra.run.smithi119.stdout: volumes: 2022-04-09T11:35:08.308 INFO:teuthology.orchestra.run.smithi119.stdout: - name: run 2022-04-09T11:35:08.308 INFO:teuthology.orchestra.run.smithi119.stdout: hostPath: 2022-04-09T11:35:08.308 INFO:teuthology.orchestra.run.smithi119.stdout: path: /run/flannel 2022-04-09T11:35:08.309 INFO:teuthology.orchestra.run.smithi119.stdout: - name: cni-plugin 2022-04-09T11:35:08.309 INFO:teuthology.orchestra.run.smithi119.stdout: hostPath: 2022-04-09T11:35:08.309 INFO:teuthology.orchestra.run.smithi119.stdout: path: /opt/cni/bin 2022-04-09T11:35:08.309 INFO:teuthology.orchestra.run.smithi119.stdout: - name: cni 2022-04-09T11:35:08.310 INFO:teuthology.orchestra.run.smithi119.stdout: hostPath: 2022-04-09T11:35:08.310 INFO:teuthology.orchestra.run.smithi119.stdout: path: /etc/cni/net.d 2022-04-09T11:35:08.310 INFO:teuthology.orchestra.run.smithi119.stdout: - name: flannel-cfg 2022-04-09T11:35:08.310 INFO:teuthology.orchestra.run.smithi119.stdout: configMap: 2022-04-09T11:35:08.310 INFO:teuthology.orchestra.run.smithi119.stdout: name: kube-flannel-cfg 2022-04-09T11:35:08.311 INFO:teuthology.orchestra.run.smithi119.stdout: - name: xtables-lock 2022-04-09T11:35:08.311 INFO:teuthology.orchestra.run.smithi119.stdout: hostPath: 2022-04-09T11:35:08.311 INFO:teuthology.orchestra.run.smithi119.stdout: path: /run/xtables.lock 2022-04-09T11:35:08.311 INFO:teuthology.orchestra.run.smithi119.stdout: type: FileOrCreate 2022-04-09T11:35:08.312 INFO:teuthology.orchestra.run.smithi119.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 5692 100 5692 0 0 106k 0 --:--:-- --:--:-- --:--:-- 108k 2022-04-09T11:35:08.339 INFO:tasks.kubeadm:Updating {'kind': 'ConfigMap', 'apiVersion': 'v1', 'metadata': {'name': 'kube-flannel-cfg', 'namespace': 'kube-system', 'labels': {'tier': 'node', 'app': 'flannel'}}, 'data': {'cni-conf.json': '{\n "name": "cbr0",\n "cniVersion": "0.3.1",\n "plugins": [\n {\n "type": "flannel",\n "delegate": {\n "hairpinMode": true,\n "isDefaultGateway": true\n }\n },\n {\n "type": "portmap",\n "capabilities": {\n "portMappings": true\n }\n }\n ]\n}\n', 'net-conf.json': '{\n "Network": "10.244.0.0/16",\n "Backend": {\n "Type": "vxlan"\n }\n}\n'}} 2022-04-09T11:35:08.340 INFO:tasks.kubeadm:Now {'kind': 'ConfigMap', 'apiVersion': 'v1', 'metadata': {'name': 'kube-flannel-cfg', 'namespace': 'kube-system', 'labels': {'tier': 'node', 'app': 'flannel'}}, 'data': {'cni-conf.json': '{\n "name": "cbr0",\n "cniVersion": "0.3.1",\n "plugins": [\n {\n "type": "flannel",\n "delegate": {\n "hairpinMode": true,\n "isDefaultGateway": true\n }\n },\n {\n "type": "portmap",\n "capabilities": {\n "portMappings": true\n }\n }\n ]\n}\n', 'net-conf.json': '{\n "Network": "10.251.176.0/21",\n "Backend": {\n "Type": "vxlan"\n }\n}\n'}} 2022-04-09T11:35:08.354 DEBUG:tasks.kubeadm:Flannel: apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: annotations: apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default name: psp.flannel.unprivileged spec: allowPrivilegeEscalation: false allowedCapabilities: - NET_ADMIN - NET_RAW allowedHostPaths: - pathPrefix: /etc/cni/net.d - pathPrefix: /etc/kube-flannel - pathPrefix: /run/flannel defaultAddCapabilities: [] defaultAllowPrivilegeEscalation: false fsGroup: rule: RunAsAny hostIPC: false hostNetwork: true hostPID: false hostPorts: - max: 65535 min: 0 privileged: false readOnlyRootFilesystem: false requiredDropCapabilities: [] runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny volumes: - configMap - secret - emptyDir - hostPath --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: flannel rules: - apiGroups: - extensions resourceNames: - psp.flannel.unprivileged resources: - podsecuritypolicies verbs: - use - apiGroups: - '' resources: - pods verbs: - get - apiGroups: - '' resources: - nodes verbs: - list - watch - apiGroups: - '' resources: - nodes/status verbs: - patch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: flannel roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: flannel subjects: - kind: ServiceAccount name: flannel namespace: kube-system --- apiVersion: v1 kind: ServiceAccount metadata: name: flannel namespace: kube-system --- apiVersion: v1 data: cni-conf.json: "{\n \"name\": \"cbr0\",\n \"cniVersion\": \"0.3.1\",\n \"plugins\"\ : [\n {\n \"type\": \"flannel\",\n \"delegate\": {\n \"hairpinMode\"\ : true,\n \"isDefaultGateway\": true\n }\n },\n {\n \"\ type\": \"portmap\",\n \"capabilities\": {\n \"portMappings\": true\n\ \ }\n }\n ]\n}\n" net-conf.json: "{\n \"Network\": \"10.251.176.0/21\",\n \"Backend\": {\n \"\ Type\": \"vxlan\"\n }\n}\n" kind: ConfigMap metadata: labels: app: flannel tier: node name: kube-flannel-cfg namespace: kube-system --- apiVersion: apps/v1 kind: DaemonSet metadata: labels: app: flannel tier: node name: kube-flannel-ds namespace: kube-system spec: selector: matchLabels: app: flannel template: metadata: labels: app: flannel tier: node spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/os operator: In values: - linux containers: - args: - --ip-masq - --kube-subnet-mgr command: - /opt/bin/flanneld env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: rancher/mirrored-flannelcni-flannel:v0.17.0 name: kube-flannel resources: limits: cpu: 100m memory: 50Mi requests: cpu: 100m memory: 50Mi securityContext: capabilities: add: - NET_ADMIN - NET_RAW privileged: false volumeMounts: - mountPath: /run/flannel name: run - mountPath: /etc/kube-flannel/ name: flannel-cfg - mountPath: /run/xtables.lock name: xtables-lock hostNetwork: true initContainers: - args: - -f - /flannel - /opt/cni/bin/flannel command: - cp image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.1 name: install-cni-plugin volumeMounts: - mountPath: /opt/cni/bin name: cni-plugin - args: - -f - /etc/kube-flannel/cni-conf.json - /etc/cni/net.d/10-flannel.conflist command: - cp image: rancher/mirrored-flannelcni-flannel:v0.17.0 name: install-cni volumeMounts: - mountPath: /etc/cni/net.d name: cni - mountPath: /etc/kube-flannel/ name: flannel-cfg priorityClassName: system-node-critical serviceAccountName: flannel tolerations: - effect: NoSchedule operator: Exists volumes: - hostPath: path: /run/flannel name: run - hostPath: path: /opt/cni/bin name: cni-plugin - hostPath: path: /etc/cni/net.d name: cni - configMap: name: kube-flannel-cfg name: flannel-cfg - hostPath: path: /run/xtables.lock type: FileOrCreate name: xtables-lock 2022-04-09T11:35:08.354 DEBUG:teuthology.orchestra.run.smithi119:> kubectl apply -f - 2022-04-09T11:35:10.737 INFO:teuthology.orchestra.run.smithi119.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-04-09T11:35:10.742 INFO:teuthology.orchestra.run.smithi119.stdout:podsecuritypolicy.policy/psp.flannel.unprivileged created 2022-04-09T11:35:10.824 INFO:teuthology.orchestra.run.smithi119.stdout:clusterrole.rbac.authorization.k8s.io/flannel created 2022-04-09T11:35:10.850 INFO:teuthology.orchestra.run.smithi119.stdout:clusterrolebinding.rbac.authorization.k8s.io/flannel created 2022-04-09T11:35:10.859 INFO:teuthology.orchestra.run.smithi119.stdout:serviceaccount/flannel created 2022-04-09T11:35:10.867 INFO:teuthology.orchestra.run.smithi119.stdout:configmap/kube-flannel-cfg created 2022-04-09T11:35:10.898 INFO:teuthology.orchestra.run.smithi119.stdout:daemonset.apps/kube-flannel-ds created 2022-04-09T11:35:10.902 INFO:tasks.kubeadm:Scanning for scratch devices 2022-04-09T11:35:10.903 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-04-09T11:35:10.903 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/scratch_devs of=/dev/stdout 2022-04-09T11:35:10.909 INFO:tasks.kubeadm:Scratch devices on smithi119: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-04-09T11:35:10.909 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-04-09T11:35:10.940 INFO:teuthology.orchestra.run.smithi119.stderr:10+0 records in 2022-04-09T11:35:10.941 INFO:teuthology.orchestra.run.smithi119.stderr:10+0 records out 2022-04-09T11:35:10.941 INFO:teuthology.orchestra.run.smithi119.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0185312 s, 566 MB/s 2022-04-09T11:35:10.942 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-04-09T11:35:10.968 INFO:teuthology.orchestra.run.smithi119.stderr:10+0 records in 2022-04-09T11:35:10.968 INFO:teuthology.orchestra.run.smithi119.stderr:10+0 records out 2022-04-09T11:35:10.968 INFO:teuthology.orchestra.run.smithi119.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0145191 s, 722 MB/s 2022-04-09T11:35:10.970 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-04-09T11:35:10.996 INFO:teuthology.orchestra.run.smithi119.stderr:10+0 records in 2022-04-09T11:35:10.996 INFO:teuthology.orchestra.run.smithi119.stderr:10+0 records out 2022-04-09T11:35:10.997 INFO:teuthology.orchestra.run.smithi119.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0140631 s, 746 MB/s 2022-04-09T11:35:10.999 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-04-09T11:35:11.070 INFO:teuthology.orchestra.run.smithi119.stderr:10+0 records in 2022-04-09T11:35:11.071 INFO:teuthology.orchestra.run.smithi119.stderr:10+0 records out 2022-04-09T11:35:11.071 INFO:teuthology.orchestra.run.smithi119.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0157794 s, 665 MB/s 2022-04-09T11:35:11.081 INFO:tasks.kubeadm:Creating PVs + StorageClass 2022-04-09T11:35:11.082 DEBUG:tasks.kubeadm:apiVersion: v1 kind: PersistentVolume metadata: name: smithi119-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi119 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi119-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi119 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi119-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi119 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi119-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi119 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: scratch provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer 2022-04-09T11:35:11.082 DEBUG:teuthology.orchestra.run.smithi119:> kubectl create -f - 2022-04-09T11:35:11.344 INFO:teuthology.orchestra.run.smithi119.stdout:persistentvolume/smithi119-lv-1 created 2022-04-09T11:35:11.364 INFO:teuthology.orchestra.run.smithi119.stdout:persistentvolume/smithi119-lv-2 created 2022-04-09T11:35:11.372 INFO:teuthology.orchestra.run.smithi119.stdout:persistentvolume/smithi119-lv-3 created 2022-04-09T11:35:11.380 INFO:teuthology.orchestra.run.smithi119.stdout:persistentvolume/smithi119-lv-4 created 2022-04-09T11:35:11.388 INFO:teuthology.orchestra.run.smithi119.stdout:storageclass.storage.k8s.io/scratch created 2022-04-09T11:35:11.396 DEBUG:teuthology.orchestra.run.smithi119:> kubectl taint node smithi119 node-role.kubernetes.io/master- || true 2022-04-09T11:35:11.490 INFO:teuthology.orchestra.run.smithi119.stdout:node/smithi119 untainted 2022-04-09T11:35:11.494 INFO:tasks.kubeadm:Kubeadm complete, yielding 2022-04-09T11:35:11.495 INFO:teuthology.run_tasks:Running task nvme_loop... 2022-04-09T11:35:11.506 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2022-04-09T11:35:11.507 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-04-09T11:35:11.507 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/scratch_devs of=/dev/stdout 2022-04-09T11:35:11.514 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-04-09T11:35:11.515 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-04-09T11:35:11.563 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-04-09T11:35:11.563 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-09T11:35:11.563 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 500 Links: 1 2022-04-09T11:35:11.564 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-09T11:35:11.564 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2022-04-09 11:35:10.946437214 +0000 2022-04-09T11:35:11.564 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2022-04-09 11:35:10.942437214 +0000 2022-04-09T11:35:11.564 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2022-04-09 11:35:10.942437214 +0000 2022-04-09T11:35:11.564 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: - 2022-04-09T11:35:11.573 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2022-04-09T11:35:11.573 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2022-04-09T11:35:11.573 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.0006601 s, 776 kB/s 2022-04-09T11:35:11.577 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-04-09T11:35:11.583 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-04-09T11:35:11.583 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-09T11:35:11.583 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 504 Links: 1 2022-04-09T11:35:11.583 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-09T11:35:11.584 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2022-04-09 11:35:10.970437215 +0000 2022-04-09T11:35:11.584 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2022-04-09 11:35:10.970437215 +0000 2022-04-09T11:35:11.584 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2022-04-09 11:35:10.970437215 +0000 2022-04-09T11:35:11.584 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: - 2022-04-09T11:35:11.590 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2022-04-09T11:35:11.590 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2022-04-09T11:35:11.590 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000533301 s, 960 kB/s 2022-04-09T11:35:11.594 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-04-09T11:35:11.600 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-04-09T11:35:11.600 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-09T11:35:11.601 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 510 Links: 1 2022-04-09T11:35:11.601 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-09T11:35:11.601 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2022-04-09 11:35:11.002437216 +0000 2022-04-09T11:35:11.601 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2022-04-09 11:35:10.998437216 +0000 2022-04-09T11:35:11.602 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2022-04-09 11:35:10.998437216 +0000 2022-04-09T11:35:11.602 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: - 2022-04-09T11:35:11.611 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2022-04-09T11:35:11.611 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2022-04-09T11:35:11.611 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.00050598 s, 1.0 MB/s 2022-04-09T11:35:11.615 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-04-09T11:35:11.622 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-04-09T11:35:11.622 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-09T11:35:11.623 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 517 Links: 1 2022-04-09T11:35:11.623 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-09T11:35:11.623 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2022-04-09 11:35:11.070437218 +0000 2022-04-09T11:35:11.623 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2022-04-09 11:35:11.070437218 +0000 2022-04-09T11:35:11.624 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2022-04-09 11:35:11.070437218 +0000 2022-04-09T11:35:11.624 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: - 2022-04-09T11:35:11.632 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2022-04-09T11:35:11.633 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2022-04-09T11:35:11.633 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000521553 s, 982 kB/s 2022-04-09T11:35:11.636 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-04-09T11:35:11.760 INFO:teuthology.orchestra.run.smithi119.stdout:loop 2022-04-09T11:35:11.762 INFO:tasks.nvme_loop:Connecting nvme_loop smithi119:/dev/vg_nvme/lv_1... 2022-04-09T11:35:11.762 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-04-09T11:35:11.785 INFO:teuthology.orchestra.run.smithi119.stdout:1 2022-04-09T11:35:11.807 INFO:teuthology.orchestra.run.smithi119.stdout:/dev/vg_nvme/lv_11 2022-04-09T11:35:11.830 INFO:tasks.nvme_loop:Connecting nvme_loop smithi119:/dev/vg_nvme/lv_2... 2022-04-09T11:35:11.830 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-04-09T11:35:11.853 INFO:teuthology.orchestra.run.smithi119.stdout:1 2022-04-09T11:35:11.877 INFO:teuthology.orchestra.run.smithi119.stdout:/dev/vg_nvme/lv_21 2022-04-09T11:35:11.900 INFO:tasks.nvme_loop:Connecting nvme_loop smithi119:/dev/vg_nvme/lv_3... 2022-04-09T11:35:11.900 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-04-09T11:35:11.964 INFO:teuthology.orchestra.run.smithi119.stdout:1 2022-04-09T11:35:11.985 INFO:teuthology.orchestra.run.smithi119.stdout:/dev/vg_nvme/lv_31 2022-04-09T11:35:12.003 INFO:tasks.nvme_loop:Connecting nvme_loop smithi119:/dev/vg_nvme/lv_4... 2022-04-09T11:35:12.003 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-04-09T11:35:12.023 INFO:teuthology.orchestra.run.smithi119.stdout:1 2022-04-09T11:35:12.043 INFO:teuthology.orchestra.run.smithi119.stdout:/dev/vg_nvme/lv_41 2022-04-09T11:35:12.067 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-04-09T11:35:12.068 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/scratch_devs of=/dev/stdout 2022-04-09T11:35:12.074 DEBUG:teuthology.orchestra.run.smithi119:> sudo nvme list 2022-04-09T11:35:12.128 INFO:teuthology.orchestra.run.smithi119.stdout:Node SN Model Namespace Usage Format FW Rev 2022-04-09T11:35:12.129 INFO:teuthology.orchestra.run.smithi119.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-04-09T11:35:12.129 INFO:teuthology.orchestra.run.smithi119.stdout:/dev/nvme0n1 PHFT620400Y1400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101F0 2022-04-09T11:35:12.129 INFO:teuthology.orchestra.run.smithi119.stdout:/dev/nvme1n1 32abb225b36e4a72 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-04-09T11:35:12.130 INFO:teuthology.orchestra.run.smithi119.stdout:/dev/nvme2n1 29fc891463a355d6 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-04-09T11:35:12.130 INFO:teuthology.orchestra.run.smithi119.stdout:/dev/nvme3n1 2d637f40c9725315 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-04-09T11:35:12.130 INFO:teuthology.orchestra.run.smithi119.stdout:/dev/nvme4n1 f1a77b1f38ed99f7 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-04-09T11:35:12.131 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-04-09T11:35:12.131 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-04-09T11:35:12.132 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/scratch_devs 2022-04-09T11:35:12.144 INFO:teuthology.run_tasks:Running task rook... 2022-04-09T11:35:12.214 INFO:tasks.rook:Rook start 2022-04-09T11:35:12.215 INFO:tasks.rook:Config: {'conf': {'global': {'osd crush chooseleaf type': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'a70e525ea98f7194c5e516757c6105ca14c81cb7', 'rook_branch': 'v1.7.2', 'rook_image': 'rook/ceph:v1.7.2'} 2022-04-09T11:35:12.215 INFO:tasks.rook:Ceph image is quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7 2022-04-09T11:35:12.216 INFO:tasks.rook:Cloning https://github.com/rook/rook branch v1.7.2 2022-04-09T11:35:12.216 DEBUG:teuthology.orchestra.run.smithi119:> rm -rf rook && git clone --single-branch --branch v1.7.2 https://github.com/rook/rook rook 2022-04-09T11:35:12.225 INFO:teuthology.orchestra.run.smithi119.stderr:Cloning into 'rook'... 2022-04-09T11:35:14.209 INFO:teuthology.orchestra.run.smithi119.stderr:Note: switching to '1784489964d2e417b6050c7edc8f626b5cca49f5'. 2022-04-09T11:35:14.209 INFO:teuthology.orchestra.run.smithi119.stderr: 2022-04-09T11:35:14.209 INFO:teuthology.orchestra.run.smithi119.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2022-04-09T11:35:14.210 INFO:teuthology.orchestra.run.smithi119.stderr:changes and commit them, and you can discard any commits you make in this 2022-04-09T11:35:14.210 INFO:teuthology.orchestra.run.smithi119.stderr:state without impacting any branches by switching back to a branch. 2022-04-09T11:35:14.210 INFO:teuthology.orchestra.run.smithi119.stderr: 2022-04-09T11:35:14.210 INFO:teuthology.orchestra.run.smithi119.stderr:If you want to create a new branch to retain commits you create, you may 2022-04-09T11:35:14.211 INFO:teuthology.orchestra.run.smithi119.stderr:do so (now or later) by using -c with the switch command. Example: 2022-04-09T11:35:14.211 INFO:teuthology.orchestra.run.smithi119.stderr: 2022-04-09T11:35:14.211 INFO:teuthology.orchestra.run.smithi119.stderr: git switch -c 2022-04-09T11:35:14.211 INFO:teuthology.orchestra.run.smithi119.stderr: 2022-04-09T11:35:14.212 INFO:teuthology.orchestra.run.smithi119.stderr:Or undo this operation with: 2022-04-09T11:35:14.212 INFO:teuthology.orchestra.run.smithi119.stderr: 2022-04-09T11:35:14.212 INFO:teuthology.orchestra.run.smithi119.stderr: git switch - 2022-04-09T11:35:14.212 INFO:teuthology.orchestra.run.smithi119.stderr: 2022-04-09T11:35:14.212 INFO:teuthology.orchestra.run.smithi119.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2022-04-09T11:35:14.213 INFO:teuthology.orchestra.run.smithi119.stderr: 2022-04-09T11:35:14.307 INFO:tasks.rook:/home/teuthworker 2022-04-09T11:35:14.307 DEBUG:teuthology.orchestra.run.smithi119:> PATH=/sbin:/usr/sbin ip addr show 2022-04-09T11:35:14.316 INFO:teuthology.orchestra.run.smithi119.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-04-09T11:35:14.316 INFO:teuthology.orchestra.run.smithi119.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-04-09T11:35:14.316 INFO:teuthology.orchestra.run.smithi119.stdout: inet 127.0.0.1/8 scope host lo 2022-04-09T11:35:14.317 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2022-04-09T11:35:14.317 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 ::1/128 scope host 2022-04-09T11:35:14.317 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2022-04-09T11:35:14.317 INFO:teuthology.orchestra.run.smithi119.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-04-09T11:35:14.317 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:62 brd ff:ff:ff:ff:ff:ff 2022-04-09T11:35:14.318 INFO:teuthology.orchestra.run.smithi119.stdout: altname enp7s0f0 2022-04-09T11:35:14.318 INFO:teuthology.orchestra.run.smithi119.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-04-09T11:35:14.318 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:63 brd ff:ff:ff:ff:ff:ff 2022-04-09T11:35:14.318 INFO:teuthology.orchestra.run.smithi119.stdout: altname enp7s0f1 2022-04-09T11:35:14.318 INFO:teuthology.orchestra.run.smithi119.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-04-09T11:35:14.319 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ee brd ff:ff:ff:ff:ff:ff 2022-04-09T11:35:14.319 INFO:teuthology.orchestra.run.smithi119.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2022-04-09T11:35:14.319 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ef brd ff:ff:ff:ff:ff:ff 2022-04-09T11:35:14.319 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.21.15.119/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2022-04-09T11:35:14.319 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft 42910sec preferred_lft 42910sec 2022-04-09T11:35:14.320 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::ec4:7aff:fe8f:d5ef/64 scope link 2022-04-09T11:35:14.320 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2022-04-09T11:35:14.320 INFO:teuthology.orchestra.run.smithi119.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2022-04-09T11:35:14.320 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 02:42:00:3c:c7:57 brd ff:ff:ff:ff:ff:ff 2022-04-09T11:35:14.321 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2022-04-09T11:35:14.321 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2022-04-09T11:35:14.322 DEBUG:teuthology.orchestra.run.smithi119:> which systemctl 2022-04-09T11:35:14.366 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/systemctl 2022-04-09T11:35:14.366 DEBUG:teuthology.orchestra.run.smithi119:> true 2022-04-09T11:35:14.399 DEBUG:teuthology.orchestra.run.smithi119:> sudo lsb_release -is 2022-04-09T11:35:14.641 INFO:teuthology.orchestra.run.smithi119.stdout:Ubuntu 2022-04-09T11:35:14.647 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-04-09T11:35:14.648 INFO:tasks.rook:['__class__', '__del__', '__delattr__', '__dir__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_format_size', '_reconnect', '_runner', '_set_iface_and_cidr', '_sftp_get_file', '_sftp_get_size', '_sftp_open_file', '_sftp_put_file', 'chcon', 'chmod', 'connect', 'copy_file', 'ensure_online', 'get_file', 'get_tar', 'get_tar_stream', 'mkdtemp', 'mktemp', 'move_file', 'put_file', 'read_file', 'reconnect', 'remove', 'run', 'sh', 'sh_file', 'sudo_write_file', 'write_file'] 2022-04-09T11:35:14.670 DEBUG:teuthology.orchestra.remote:smithi119:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-04-09T11:35:14.685 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-04-09T11:35:14.686 DEBUG:teuthology.orchestra.run.smithi119:> dd if=rook/cluster/examples/kubernetes/ceph/operator.yaml of=/dev/stdout 2022-04-09T11:35:14.695 INFO:tasks.rook:Patching operator to use image rook/ceph:v1.7.2 2022-04-09T11:35:14.734 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-04-09T11:35:14.734 DEBUG:teuthology.orchestra.run.smithi119:> dd of=operator.yaml 2022-04-09T11:35:14.746 INFO:tasks.rook:Deploying operator 2022-04-09T11:35:14.765 DEBUG:teuthology.orchestra.remote:smithi119:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-04-09T11:35:14.842 DEBUG:teuthology.orchestra.remote:smithi119:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-04-09T11:35:14.857 DEBUG:teuthology.orchestra.run.smithi119:> kubectl create -f rook/cluster/examples/kubernetes/ceph/crds.yaml -f rook/cluster/examples/kubernetes/ceph/common.yaml -f operator.yaml 2022-04-09T11:35:15.120 INFO:teuthology.orchestra.run.smithi119.stdout:customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created 2022-04-09T11:35:15.135 INFO:teuthology.orchestra.run.smithi119.stdout:customresourcedefinition.apiextensions.k8s.io/cephclients.ceph.rook.io created 2022-04-09T11:35:15.253 INFO:teuthology.orchestra.run.smithi119.stdout:customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created 2022-04-09T11:35:15.279 INFO:teuthology.orchestra.run.smithi119.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemmirrors.ceph.rook.io created 2022-04-09T11:35:15.312 INFO:teuthology.orchestra.run.smithi119.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created 2022-04-09T11:35:15.408 INFO:teuthology.orchestra.run.smithi119.stdout:customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created 2022-04-09T11:35:15.430 INFO:teuthology.orchestra.run.smithi119.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectrealms.ceph.rook.io created 2022-04-09T11:35:15.482 INFO:teuthology.orchestra.run.smithi119.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created 2022-04-09T11:35:15.506 INFO:teuthology.orchestra.run.smithi119.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created 2022-04-09T11:35:15.620 INFO:teuthology.orchestra.run.smithi119.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzonegroups.ceph.rook.io created 2022-04-09T11:35:15.635 INFO:teuthology.orchestra.run.smithi119.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzones.ceph.rook.io created 2022-04-09T11:35:15.668 INFO:teuthology.orchestra.run.smithi119.stdout:customresourcedefinition.apiextensions.k8s.io/cephrbdmirrors.ceph.rook.io created 2022-04-09T11:35:15.686 INFO:teuthology.orchestra.run.smithi119.stdout:customresourcedefinition.apiextensions.k8s.io/objectbucketclaims.objectbucket.io created 2022-04-09T11:35:15.712 INFO:teuthology.orchestra.run.smithi119.stdout:customresourcedefinition.apiextensions.k8s.io/objectbuckets.objectbucket.io created 2022-04-09T11:35:15.803 INFO:teuthology.orchestra.run.smithi119.stdout:customresourcedefinition.apiextensions.k8s.io/volumereplicationclasses.replication.storage.openshift.io created 2022-04-09T11:35:15.829 INFO:teuthology.orchestra.run.smithi119.stdout:customresourcedefinition.apiextensions.k8s.io/volumereplications.replication.storage.openshift.io created 2022-04-09T11:35:15.855 INFO:teuthology.orchestra.run.smithi119.stdout:customresourcedefinition.apiextensions.k8s.io/volumes.rook.io created 2022-04-09T11:35:15.977 INFO:teuthology.orchestra.run.smithi119.stdout:namespace/rook-ceph created 2022-04-09T11:35:16.004 INFO:teuthology.orchestra.run.smithi119.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-04-09T11:35:16.029 INFO:teuthology.orchestra.run.smithi119.stdout:serviceaccount/rook-ceph-admission-controller created 2022-04-09T11:35:16.054 INFO:teuthology.orchestra.run.smithi119.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-admission-controller-role created 2022-04-09T11:35:16.161 INFO:teuthology.orchestra.run.smithi119.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-admission-controller-rolebinding created 2022-04-09T11:35:16.165 INFO:teuthology.orchestra.run.smithi119.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-04-09T11:35:16.169 INFO:teuthology.orchestra.run.smithi119.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-system created 2022-04-09T11:35:16.196 INFO:teuthology.orchestra.run.smithi119.stdout:role.rbac.authorization.k8s.io/rook-ceph-system created 2022-04-09T11:35:16.203 INFO:teuthology.orchestra.run.smithi119.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-global created 2022-04-09T11:35:16.211 INFO:teuthology.orchestra.run.smithi119.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-04-09T11:35:16.237 INFO:teuthology.orchestra.run.smithi119.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-04-09T11:35:16.244 INFO:teuthology.orchestra.run.smithi119.stdout:serviceaccount/rook-ceph-system created 2022-04-09T11:35:16.253 INFO:teuthology.orchestra.run.smithi119.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-04-09T11:35:16.304 INFO:teuthology.orchestra.run.smithi119.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-04-09T11:35:16.307 INFO:teuthology.orchestra.run.smithi119.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created 2022-04-09T11:35:16.311 INFO:teuthology.orchestra.run.smithi119.stdout:serviceaccount/rook-ceph-osd created 2022-04-09T11:35:16.319 INFO:teuthology.orchestra.run.smithi119.stdout:serviceaccount/rook-ceph-mgr created 2022-04-09T11:35:16.346 INFO:teuthology.orchestra.run.smithi119.stdout:serviceaccount/rook-ceph-cmd-reporter created 2022-04-09T11:35:16.354 INFO:teuthology.orchestra.run.smithi119.stdout:role.rbac.authorization.k8s.io/rook-ceph-osd created 2022-04-09T11:35:16.380 INFO:teuthology.orchestra.run.smithi119.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-osd created 2022-04-09T11:35:16.452 INFO:teuthology.orchestra.run.smithi119.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-04-09T11:35:16.456 INFO:teuthology.orchestra.run.smithi119.stdout:role.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-04-09T11:35:16.479 INFO:teuthology.orchestra.run.smithi119.stdout:role.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-04-09T11:35:16.487 INFO:teuthology.orchestra.run.smithi119.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-04-09T11:35:16.494 INFO:teuthology.orchestra.run.smithi119.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-04-09T11:35:16.503 INFO:teuthology.orchestra.run.smithi119.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-04-09T11:35:16.511 INFO:teuthology.orchestra.run.smithi119.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-04-09T11:35:16.520 INFO:teuthology.orchestra.run.smithi119.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-04-09T11:35:16.528 INFO:teuthology.orchestra.run.smithi119.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-04-09T11:35:16.601 INFO:teuthology.orchestra.run.smithi119.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-04-09T11:35:16.606 INFO:teuthology.orchestra.run.smithi119.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-04-09T11:35:16.607 INFO:teuthology.orchestra.run.smithi119.stdout:podsecuritypolicy.policy/00-rook-privileged created 2022-04-09T11:35:16.611 INFO:teuthology.orchestra.run.smithi119.stdout:clusterrole.rbac.authorization.k8s.io/psp:rook created 2022-04-09T11:35:16.620 INFO:teuthology.orchestra.run.smithi119.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system-psp created 2022-04-09T11:35:16.628 INFO:teuthology.orchestra.run.smithi119.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-default-psp created 2022-04-09T11:35:16.637 INFO:teuthology.orchestra.run.smithi119.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd-psp created 2022-04-09T11:35:16.663 INFO:teuthology.orchestra.run.smithi119.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-psp created 2022-04-09T11:35:16.670 INFO:teuthology.orchestra.run.smithi119.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter-psp created 2022-04-09T11:35:16.700 INFO:teuthology.orchestra.run.smithi119.stdout:serviceaccount/rook-csi-cephfs-plugin-sa created 2022-04-09T11:35:16.751 INFO:teuthology.orchestra.run.smithi119.stdout:serviceaccount/rook-csi-cephfs-provisioner-sa created 2022-04-09T11:35:16.755 INFO:teuthology.orchestra.run.smithi119.stdout:role.rbac.authorization.k8s.io/cephfs-external-provisioner-cfg created 2022-04-09T11:35:16.761 INFO:teuthology.orchestra.run.smithi119.stdout:rolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role-cfg created 2022-04-09T11:35:16.788 INFO:teuthology.orchestra.run.smithi119.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-04-09T11:35:16.795 INFO:teuthology.orchestra.run.smithi119.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-external-provisioner-runner created 2022-04-09T11:35:16.821 INFO:teuthology.orchestra.run.smithi119.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-plugin-sa-psp created 2022-04-09T11:35:16.828 INFO:teuthology.orchestra.run.smithi119.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-provisioner-sa-psp created 2022-04-09T11:35:17.051 INFO:teuthology.orchestra.run.smithi119.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-04-09T11:35:17.053 INFO:teuthology.orchestra.run.smithi119.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role created 2022-04-09T11:35:17.120 INFO:teuthology.orchestra.run.smithi119.stdout:serviceaccount/rook-csi-rbd-plugin-sa created 2022-04-09T11:35:17.335 INFO:teuthology.orchestra.run.smithi119.stdout:serviceaccount/rook-csi-rbd-provisioner-sa created 2022-04-09T11:35:17.370 INFO:teuthology.orchestra.run.smithi119.stdout:role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created 2022-04-09T11:35:17.398 INFO:teuthology.orchestra.run.smithi119.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created 2022-04-09T11:35:17.423 INFO:teuthology.orchestra.run.smithi119.stdout:clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-04-09T11:35:17.494 INFO:teuthology.orchestra.run.smithi119.stdout:clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created 2022-04-09T11:35:17.498 INFO:teuthology.orchestra.run.smithi119.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-plugin-sa-psp created 2022-04-09T11:35:17.503 INFO:teuthology.orchestra.run.smithi119.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-provisioner-sa-psp created 2022-04-09T11:35:17.533 INFO:teuthology.orchestra.run.smithi119.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-04-09T11:35:17.556 INFO:teuthology.orchestra.run.smithi119.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created 2022-04-09T11:35:17.581 INFO:teuthology.orchestra.run.smithi119.stdout:role.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-04-09T11:35:17.586 INFO:teuthology.orchestra.run.smithi119.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-04-09T11:35:17.643 INFO:teuthology.orchestra.run.smithi119.stdout:serviceaccount/rook-ceph-purge-osd created 2022-04-09T11:35:17.653 INFO:teuthology.orchestra.run.smithi119.stdout:configmap/rook-ceph-operator-config created 2022-04-09T11:35:17.682 INFO:teuthology.orchestra.run.smithi119.stdout:deployment.apps/rook-ceph-operator created 2022-04-09T11:35:17.686 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-04-09T11:35:17.740 INFO:teuthology.orchestra.run.smithi119.stderr:No resources found in rook-ceph namespace. 2022-04-09T11:35:27.744 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-04-09T11:35:27.818 INFO:teuthology.orchestra.run.smithi119.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:35:27.818 INFO:teuthology.orchestra.run.smithi119.stdout:rook-ceph-operator-6b496c8959-sbdxf 0/1 Pending 0 6s 2022-04-09T11:35:37.820 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-04-09T11:35:37.885 INFO:teuthology.orchestra.run.smithi119.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:35:37.885 INFO:teuthology.orchestra.run.smithi119.stdout:rook-ceph-operator-6b496c8959-sbdxf 0/1 Pending 0 16s 2022-04-09T11:35:47.887 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-04-09T11:35:47.949 INFO:teuthology.orchestra.run.smithi119.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:35:47.950 INFO:teuthology.orchestra.run.smithi119.stdout:rook-ceph-operator-6b496c8959-sbdxf 0/1 ContainerCreating 0 26s 2022-04-09T11:35:57.952 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-04-09T11:35:58.019 INFO:teuthology.orchestra.run.smithi119.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:35:58.019 INFO:teuthology.orchestra.run.smithi119.stdout:rook-ceph-operator-6b496c8959-sbdxf 0/1 ContainerCreating 0 37s 2022-04-09T11:36:08.022 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-04-09T11:36:08.572 INFO:teuthology.orchestra.run.smithi119.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:36:08.573 INFO:teuthology.orchestra.run.smithi119.stdout:rook-ceph-operator-6b496c8959-sbdxf 0/1 ContainerCreating 0 47s 2022-04-09T11:36:18.576 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-04-09T11:36:18.639 INFO:teuthology.orchestra.run.smithi119.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:36:18.639 INFO:teuthology.orchestra.run.smithi119.stdout:rook-ceph-operator-6b496c8959-sbdxf 1/1 Running 0 57s 2022-04-09T11:36:18.641 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph logs -f rook-ceph-operator-6b496c8959-sbdxf 2022-04-09T11:36:18.649 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-04-09T11:36:18.650 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/scratch_devs of=/dev/stdout 2022-04-09T11:36:18.700 INFO:tasks.rook: override: [global] osd crush chooseleaf type = 0 2022-04-09T11:36:18.700 INFO:tasks.rook: override: [mgr] debug mgr = 20 2022-04-09T11:36:18.701 INFO:tasks.rook: override: [mgr] debug ms = 1 2022-04-09T11:36:18.701 INFO:tasks.rook: override: [mon] debug mon = 20 2022-04-09T11:36:18.701 INFO:tasks.rook: override: [mon] debug ms = 1 2022-04-09T11:36:18.701 INFO:tasks.rook: override: [mon] debug paxos = 20 2022-04-09T11:36:18.702 INFO:tasks.rook: override: [osd] debug ms = 1 2022-04-09T11:36:18.702 INFO:tasks.rook: override: [osd] debug osd = 20 2022-04-09T11:36:18.703 INFO:tasks.rook:Config: b"[global]\n\nlog to file = true\n\nmon clock drift allowed = 1.000\n\n# replicate across OSDs, not hosts\nosd crush chooseleaf type = 0\n\n# enable some debugging\nauth debug = true\nms die on old message = true\nms die on bug = true\ndebug asserts on shutdown = true\n\n\n[osd]\n# debugging\nosd debug shutdown = true\nosd debug op order = true\nosd debug verify stray on activate = true\nosd debug pg log writeout = true\nosd debug verify cached snaps = true\nosd debug verify missing on start = true\nosd debug misdirected ops = true\nosd op queue = debug_random\nosd op queue cut off = debug_random\nosd shutdown pgref assert = true\nbdev debug aio = true\nosd sloppy crc = true\ndebug ms = 1\ndebug osd = 20\n\n\n[mon]\n# rotate auth tickets quickly to exercise renewal paths\nauth mon ticket ttl = 660# 11m\nauth service ticket ttl = 240# 4m\n\n# don't complain about global id reclaim\nmon_warn_on_insecure_global_id_reclaim = false\nmon_warn_on_insecure_global_id_reclaim_allowed = false\ndebug mon = 20\ndebug ms = 1\ndebug paxos = 20\n[mgr]\ndebug mgr = 20\ndebug ms = 1\n" 2022-04-09T11:36:18.705 DEBUG:teuthology.orchestra.run.smithi119:> kubectl create -f - 2022-04-09T11:36:18.745 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:12.687334 I | rookcmd: starting Rook v1.7.2 with arguments '/usr/local/bin/rook ceph operator' 2022-04-09T11:36:18.745 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:12.687417 I | rookcmd: flag values: --add_dir_header=false, --alsologtostderr=false, --csi-cephfs-plugin-template-path=/etc/ceph-csi/cephfs/csi-cephfsplugin.yaml, --csi-cephfs-provisioner-dep-template-path=/etc/ceph-csi/cephfs/csi-cephfsplugin-provisioner-dep.yaml, --csi-rbd-plugin-template-path=/etc/ceph-csi/rbd/csi-rbdplugin.yaml, --csi-rbd-provisioner-dep-template-path=/etc/ceph-csi/rbd/csi-rbdplugin-provisioner-dep.yaml, --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-flush-frequency=5s, --log-level=INFO, --log_backtrace_at=:0, --log_dir=, --log_file=, --log_file_max_size=1800, --logtostderr=true, --one_output=false, --operator-image=, --service-account=, --skip_headers=false, --skip_log_headers=false, --stderrthreshold=2, --v=0, --vmodule= 2022-04-09T11:36:18.746 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:12.687420 I | cephcmd: starting Rook-Ceph operator 2022-04-09T11:36:18.746 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:12.832115 I | cephcmd: base ceph version inside the rook operator image is "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)" 2022-04-09T11:36:18.750 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:12.839563 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-04-09T11:36:18.750 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:12.843298 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-04-09T11:36:18.750 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:12.845863 I | operator: looking for secret "rook-ceph-admission-controller" 2022-04-09T11:36:18.750 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:12.847349 I | operator: secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-04-09T11:36:18.751 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:12.849482 I | op-k8sutil: ROOK_ENABLE_FLEX_DRIVER="false" (configmap) 2022-04-09T11:36:18.751 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:12.849500 I | operator: watching all namespaces for ceph cluster CRs 2022-04-09T11:36:18.751 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:12.849595 I | operator: setting up the controller-runtime manager 2022-04-09T11:36:18.751 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:12.851371 I | ceph-cluster-controller: ConfigMap "rook-ceph-operator-config" changes detected. Updating configurations 2022-04-09T11:36:18.751 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:12.857181 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-04-09T11:36:18.752 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:13.156250 I | ceph-cluster-controller: successfully started 2022-04-09T11:36:18.752 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:13.156326 I | ceph-cluster-controller: enabling hotplug orchestration 2022-04-09T11:36:18.752 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:13.156357 I | ceph-crashcollector-controller: successfully started 2022-04-09T11:36:18.752 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:13.156434 I | ceph-block-pool-controller: successfully started 2022-04-09T11:36:18.752 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:13.156507 I | ceph-object-store-user-controller: successfully started 2022-04-09T11:36:18.753 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:13.156575 I | ceph-object-realm-controller: successfully started 2022-04-09T11:36:18.753 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:13.156640 I | ceph-object-zonegroup-controller: successfully started 2022-04-09T11:36:18.753 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:13.156720 I | ceph-object-zone-controller: successfully started 2022-04-09T11:36:18.753 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:13.156873 I | ceph-object-controller: successfully started 2022-04-09T11:36:18.754 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:13.156959 I | ceph-file-controller: successfully started 2022-04-09T11:36:18.754 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:13.157049 I | ceph-nfs-controller: successfully started 2022-04-09T11:36:18.754 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:13.157146 I | ceph-rbd-mirror-controller: successfully started 2022-04-09T11:36:18.754 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:13.157247 I | ceph-client-controller: successfully started 2022-04-09T11:36:18.754 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:13.157315 I | ceph-filesystem-mirror-controller: successfully started 2022-04-09T11:36:18.755 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:13.158549 I | operator: starting the controller-runtime manager 2022-04-09T11:36:19.497 INFO:teuthology.orchestra.run.smithi119.stdout:configmap/rook-config-override created 2022-04-09T11:36:19.501 INFO:tasks.rook:Cluster: apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: allowUnsupported: true image: quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7 dataDirHostPath: /var/lib/rook mgr: count: 1 modules: - enabled: true name: rook mon: allowMultiplePerNode: true count: 1 skipUpgradeChecks: true 2022-04-09T11:36:19.502 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-04-09T11:36:19.502 DEBUG:teuthology.orchestra.run.smithi119:> dd of=cluster.yaml 2022-04-09T11:36:19.508 DEBUG:teuthology.orchestra.run.smithi119:> kubectl create -f cluster.yaml 2022-04-09T11:36:21.799 INFO:teuthology.orchestra.run.smithi119.stdout:cephcluster.ceph.rook.io/rook-ceph created 2022-04-09T11:36:21.805 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:21.804346 I | clusterdisruption-controller: create event from ceph cluster CR 2022-04-09T11:36:21.805 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:21.804544 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "rook-ceph" 2022-04-09T11:36:21.852 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:21.851893 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-04-09T11:36:21.854 DEBUG:teuthology.orchestra.remote:smithi119:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-04-09T11:36:21.856 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:21.853464 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-04-09T11:36:21.856 INFO:tasks.rook.operator.smithi119.stdout: ... // 3 identical fields 2022-04-09T11:36:21.857 INFO:tasks.rook.operator.smithi119.stdout: Labels: nil, 2022-04-09T11:36:21.857 INFO:tasks.rook.operator.smithi119.stdout: Placement: nil, 2022-04-09T11:36:21.857 INFO:tasks.rook.operator.smithi119.stdout: Network: v1.NetworkSpec{ 2022-04-09T11:36:21.858 INFO:tasks.rook.operator.smithi119.stdout: Provider: "", 2022-04-09T11:36:21.858 INFO:tasks.rook.operator.smithi119.stdout: Selectors: nil, 2022-04-09T11:36:21.858 INFO:tasks.rook.operator.smithi119.stdout: HostNetwork: false, 2022-04-09T11:36:21.858 INFO:tasks.rook.operator.smithi119.stdout:- IPFamily: "", 2022-04-09T11:36:21.859 INFO:tasks.rook.operator.smithi119.stdout:+ IPFamily: "IPv4", 2022-04-09T11:36:21.859 INFO:tasks.rook.operator.smithi119.stdout: DualStack: false, 2022-04-09T11:36:21.860 INFO:tasks.rook.operator.smithi119.stdout: }, 2022-04-09T11:36:21.860 INFO:tasks.rook.operator.smithi119.stdout: Resources: nil, 2022-04-09T11:36:21.860 INFO:tasks.rook.operator.smithi119.stdout: PriorityClassNames: nil, 2022-04-09T11:36:21.861 INFO:tasks.rook.operator.smithi119.stdout: ... // 16 identical fields 2022-04-09T11:36:21.861 INFO:tasks.rook.operator.smithi119.stdout: } 2022-04-09T11:36:21.862 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:21.854654 I | op-k8sutil: ROOK_ENABLE_FLEX_DRIVER="false" (configmap) 2022-04-09T11:36:21.867 DEBUG:teuthology.orchestra.run.smithi119:> kubectl create -f rook/cluster/examples/kubernetes/ceph/toolbox.yaml 2022-04-09T11:36:21.886 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:21.885920 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-04-09T11:36:21.896 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:21.896865 I | ceph-cluster-controller: clusterInfo not yet found, must be a new cluster 2022-04-09T11:36:21.897 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:21.896905 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-04-09T11:36:21.900 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:21.899264 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-04-09T11:36:21.903 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:21.901280 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-04-09T11:36:21.903 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:21.903647 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-04-09T11:36:21.909 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:21.905597 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-04-09T11:36:21.911 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:21.910478 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.4.0" (default) 2022-04-09T11:36:21.914 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:21.912481 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7... 2022-04-09T11:36:22.055 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:22.055430 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0" (default) 2022-04-09T11:36:22.100 INFO:teuthology.orchestra.run.smithi119.stdout:deployment.apps/rook-ceph-tools created 2022-04-09T11:36:22.106 INFO:tasks.rook:Waiting for tools container to start 2022-04-09T11:36:22.106 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-09T11:36:22.219 INFO:teuthology.orchestra.run.smithi119.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:36:22.219 INFO:teuthology.orchestra.run.smithi119.stdout:rook-ceph-tools-7677f7f4b9-69kwt 0/1 Pending 0 0s 2022-04-09T11:36:22.456 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:22.456213 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v2.2.2" (default) 2022-04-09T11:36:23.170 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:23.170179 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.2.1" (default) 2022-04-09T11:36:23.327 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:23.325365 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1" (default) 2022-04-09T11:36:23.330 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:23.328141 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-04-09T11:36:23.456 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:23.456398 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.1.0" (default) 2022-04-09T11:36:23.655 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:23.655447 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-04-09T11:36:23.856 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:23.856157 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-04-09T11:36:23.857 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:23.856191 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.4.0" 2022-04-09T11:36:24.057 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:36:24.057087 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-04-09T11:36:27.224 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-09T11:36:27.341 INFO:teuthology.orchestra.run.smithi119.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:36:27.342 INFO:teuthology.orchestra.run.smithi119.stdout:rook-ceph-tools-7677f7f4b9-69kwt 0/1 ContainerCreating 0 5s 2022-04-09T11:36:32.346 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-09T11:36:32.573 INFO:teuthology.orchestra.run.smithi119.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:36:32.574 INFO:teuthology.orchestra.run.smithi119.stdout:rook-ceph-tools-7677f7f4b9-69kwt 0/1 ContainerCreating 0 10s 2022-04-09T11:36:37.576 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-09T11:36:37.641 INFO:teuthology.orchestra.run.smithi119.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:36:37.641 INFO:teuthology.orchestra.run.smithi119.stdout:rook-ceph-tools-7677f7f4b9-69kwt 0/1 ContainerCreating 0 15s 2022-04-09T11:36:42.643 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-09T11:36:42.712 INFO:teuthology.orchestra.run.smithi119.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:36:42.712 INFO:teuthology.orchestra.run.smithi119.stdout:rook-ceph-tools-7677f7f4b9-69kwt 0/1 ContainerCreating 0 20s 2022-04-09T11:36:47.715 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-09T11:36:47.780 INFO:teuthology.orchestra.run.smithi119.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:36:47.780 INFO:teuthology.orchestra.run.smithi119.stdout:rook-ceph-tools-7677f7f4b9-69kwt 0/1 ContainerCreating 0 25s 2022-04-09T11:36:52.783 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-09T11:36:52.840 INFO:teuthology.orchestra.run.smithi119.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:36:52.840 INFO:teuthology.orchestra.run.smithi119.stdout:rook-ceph-tools-7677f7f4b9-69kwt 0/1 ContainerCreating 0 30s 2022-04-09T11:36:57.843 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-09T11:36:57.911 INFO:teuthology.orchestra.run.smithi119.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:36:57.911 INFO:teuthology.orchestra.run.smithi119.stdout:rook-ceph-tools-7677f7f4b9-69kwt 0/1 ContainerCreating 0 35s 2022-04-09T11:37:02.749 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:02.742903 I | ceph-csi: Detected ceph CSI image version: "v3.4.0" 2022-04-09T11:37:02.791 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:02.792528 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-04-09T11:37:02.914 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-09T11:37:03.240 INFO:teuthology.orchestra.run.smithi119.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:37:03.240 INFO:teuthology.orchestra.run.smithi119.stdout:rook-ceph-tools-7677f7f4b9-69kwt 0/1 ContainerCreating 0 41s 2022-04-09T11:37:03.241 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:03.240434 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-04-09T11:37:03.243 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:03.243831 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-04-09T11:37:03.246 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:03.246412 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-04-09T11:37:03.470 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:03.469693 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-04-09T11:37:03.509 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:03.510607 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-04-09T11:37:03.680 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:03.680375 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-04-09T11:37:03.683 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:03.682019 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11488 quincy" 2022-04-09T11:37:03.683 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:03.682058 I | ceph-cluster-controller: validating ceph version from provided image 2022-04-09T11:37:03.683 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:03.682073 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11488 quincy", pursuing 2022-04-09T11:37:03.723 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:03.723894 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11488 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7" 2022-04-09T11:37:03.723 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:03.723943 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-04-09T11:37:03.745 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:03.746341 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-04-09T11:37:03.857 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:03.857973 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-04-09T11:37:03.922 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:03.923137 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-04-09T11:37:04.160 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:04.161481 I | op-mon: start running mons 2022-04-09T11:37:04.330 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:04.331477 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-04-09T11:37:04.569 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:04.570934 I | op-mon: creating mon secrets for a new cluster 2022-04-09T11:37:04.715 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:04.716288 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-04-09T11:37:04.715 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:04.716316 I | ceph-csi: Kubernetes version is 1.23 2022-04-09T11:37:05.117 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:05.118227 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.2.0" (default) 2022-04-09T11:37:05.516 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:05.517196 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-04-09T11:37:05.743 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:05.744024 I | op-mon: existing maxMonID not found or failed to load. configmaps "rook-ceph-mon-endpoints" not found 2022-04-09T11:37:06.117 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:06.119140 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[]}] data: mapping:{"node":{}} maxMonId:-1] 2022-04-09T11:37:06.315 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:06.317155 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-04-09T11:37:06.715 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:06.716819 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-04-09T11:37:07.117 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:07.119669 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-04-09T11:37:07.515 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:07.516906 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-04-09T11:37:07.715 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:07.717378 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-09T11:37:07.716 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:07.717605 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-09T11:37:07.915 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:07.916667 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-04-09T11:37:08.245 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-09T11:37:08.314 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:08.315807 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-04-09T11:37:08.315 INFO:teuthology.orchestra.run.smithi119.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:37:08.315 INFO:teuthology.orchestra.run.smithi119.stdout:rook-ceph-tools-7677f7f4b9-69kwt 0/1 ContainerCreating 0 46s 2022-04-09T11:37:08.714 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:08.716499 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-04-09T11:37:09.115 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:09.116897 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-04-09T11:37:09.693 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:09.695462 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-04-09T11:37:09.915 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:09.917529 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-04-09T11:37:09.931 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:09.933476 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-04-09T11:37:10.336 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:10.338546 I | op-mon: targeting the mon count 1 2022-04-09T11:37:10.562 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:10.564005 I | ceph-cluster-controller: Cluster update requested. CANCELLING CURRENT ORCHESTRATION 2022-04-09T11:37:10.562 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:10.564182 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-04-09T11:37:10.565 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:10.564509 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-04-09T11:37:11.055 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:11.057192 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-04-09T11:37:11.126 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:11.125960 I | op-mon: parsing mon endpoints: 2022-04-09T11:37:11.127 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:11.125989 W | op-mon: ignoring invalid monitor 2022-04-09T11:37:11.132 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:11.134001 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7... 2022-04-09T11:37:11.337 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:11.339140 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-04-09T11:37:11.977 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:11.978488 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-04-09T11:37:12.217 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:12.219430 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-04-09T11:37:12.650 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:12.652207 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-04-09T11:37:12.660 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:12.662271 I | ceph-csi: successfully started CSI CephFS driver 2022-04-09T11:37:13.317 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-09T11:37:13.469 INFO:teuthology.orchestra.run.smithi119.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:37:13.469 INFO:teuthology.orchestra.run.smithi119.stdout:rook-ceph-tools-7677f7f4b9-69kwt 0/1 ContainerCreating 0 51s 2022-04-09T11:37:13.470 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:13.468172 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-04-09T11:37:13.553 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:13.555043 I | ceph-csi: CSIDriver object created for driver "rook-ceph.rbd.csi.ceph.com" 2022-04-09T11:37:13.690 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:13.687505 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="None" (configmap) 2022-04-09T11:37:13.696 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:13.693599 I | ceph-csi: CSIDriver object created for driver "rook-ceph.cephfs.csi.ceph.com" 2022-04-09T11:37:18.473 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-09T11:37:18.624 INFO:teuthology.orchestra.run.smithi119.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:37:18.624 INFO:teuthology.orchestra.run.smithi119.stdout:rook-ceph-tools-7677f7f4b9-69kwt 0/1 ContainerCreating 0 56s 2022-04-09T11:37:23.626 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-09T11:37:23.866 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:23.867699 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11488 quincy" 2022-04-09T11:37:23.866 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:23.867725 I | ceph-cluster-controller: validating ceph version from provided image 2022-04-09T11:37:23.867 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:23.867740 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11488 quincy", pursuing 2022-04-09T11:37:23.867 INFO:teuthology.orchestra.run.smithi119.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:37:23.868 INFO:teuthology.orchestra.run.smithi119.stdout:rook-ceph-tools-7677f7f4b9-69kwt 0/1 ContainerCreating 0 61s 2022-04-09T11:37:24.068 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:24.070309 I | op-mon: parsing mon endpoints: 2022-04-09T11:37:24.069 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:24.070345 W | op-mon: ignoring invalid monitor 2022-04-09T11:37:24.387 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:24.306962 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-09T11:37:24.392 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:24.390505 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-09T11:37:24.492 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:24.494136 E | ceph-cluster-controller: failed to get ceph daemons versions, this typically happens during the first cluster initialization. failed to run 'ceph versions'. . unable to get monitor info from DNS SRV with service name: ceph-mon 2022-04-09T11:37:24.492 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T11:37:24.487+0000 7f2ef32bb700 -1 failed for service _ceph-mon._tcp 2022-04-09T11:37:24.492 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T11:37:24.487+0000 7f2ef32bb700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-04-09T11:37:24.493 INFO:tasks.rook.operator.smithi119.stdout:[errno 2] RADOS object not found (error connecting to the cluster): exit status 1 2022-04-09T11:37:24.493 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:24.494159 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11488 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7" 2022-04-09T11:37:25.188 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:25.190939 I | op-mon: start running mons 2022-04-09T11:37:25.345 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:25.347190 I | op-mon: parsing mon endpoints: 2022-04-09T11:37:25.345 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:25.347212 W | op-mon: ignoring invalid monitor 2022-04-09T11:37:25.460 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:25.462999 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[]}] data: mapping:{"node":{}} maxMonId:-1] 2022-04-09T11:37:25.477 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:25.479506 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-09T11:37:25.477 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:25.479701 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-09T11:37:25.994 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:25.996272 I | op-mon: targeting the mon count 1 2022-04-09T11:37:25.999 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:26.001670 I | op-mon: sched-mon: created canary deployment rook-ceph-mon-a-canary 2022-04-09T11:37:27.270 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:27.271261 I | op-mon: sched-mon: canary monitor deployment rook-ceph-mon-a-canary scheduled to smithi119 2022-04-09T11:37:27.270 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:27.271286 I | op-mon: assignmon: mon a assigned to node smithi119 2022-04-09T11:37:28.073 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:28.073333 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-a-canary" 2022-04-09T11:37:28.412 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:28.414619 I | op-mon: creating mon a 2022-04-09T11:37:28.870 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-09T11:37:28.997 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:28.999382 I | op-mon: mon "a" endpoint is [v2:10.107.24.44:3300,v1:10.107.24.44:6789] 2022-04-09T11:37:29.238 INFO:teuthology.orchestra.run.smithi119.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:37:29.239 INFO:teuthology.orchestra.run.smithi119.stdout:rook-ceph-tools-7677f7f4b9-69kwt 0/1 ContainerCreating 0 67s 2022-04-09T11:37:29.773 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:29.776029 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-04-09T11:37:29.774 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:29.776095 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.24.44:6789"]}] data:a=10.107.24.44:6789 mapping:{"node":{"a":{"Name":"smithi119","Hostname":"smithi119","Address":"172.21.15.119"}}} maxMonId:-1] 2022-04-09T11:37:29.774 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:29.776219 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-04-09T11:37:30.220 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:30.222800 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-09T11:37:30.239 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:30.242293 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-09T11:37:30.820 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:30.820665 I | op-mon: 0 of 1 expected mons are ready. creating or updating deployments without checking quorum in attempt to achieve a healthy mon cluster 2022-04-09T11:37:31.255 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:31.254784 I | op-mon: updating maxMonID from -1 to 0 after committing mon "a" 2022-04-09T11:37:33.075 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:33.078057 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.24.44:6789"]}] data:a=10.107.24.44:6789 mapping:{"node":{"a":{"Name":"smithi119","Hostname":"smithi119","Address":"172.21.15.119"}}} maxMonId:0] 2022-04-09T11:37:33.076 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:33.078094 I | op-mon: waiting for mon quorum with [a] 2022-04-09T11:37:33.563 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:33.565720 I | op-mon: mon a is not yet running 2022-04-09T11:37:33.563 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:33.565741 I | op-mon: mons running: [] 2022-04-09T11:37:34.241 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-09T11:37:34.301 INFO:teuthology.orchestra.run.smithi119.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:37:34.301 INFO:teuthology.orchestra.run.smithi119.stdout:rook-ceph-tools-7677f7f4b9-69kwt 0/1 ContainerCreating 0 72s 2022-04-09T11:37:39.311 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-04-09T11:37:39.641 INFO:teuthology.orchestra.run.smithi119.stdout:NAME READY STATUS RESTARTS AGE 2022-04-09T11:37:39.642 INFO:teuthology.orchestra.run.smithi119.stdout:rook-ceph-tools-7677f7f4b9-69kwt 1/1 Running 0 77s 2022-04-09T11:37:39.643 INFO:tasks.rook:Waiting for mgr/rook orchestrator to be available 2022-04-09T11:37:39.644 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch status -f json 2022-04-09T11:37:39.908 INFO:teuthology.orchestra.run.smithi119.stderr:unable to get monitor info from DNS SRV with service name: ceph-mon 2022-04-09T11:37:39.908 INFO:teuthology.orchestra.run.smithi119.stderr:2022-04-09T11:37:39.907+0000 7fcd0d632700 -1 failed for service _ceph-mon._tcp 2022-04-09T11:37:39.909 INFO:teuthology.orchestra.run.smithi119.stderr:2022-04-09T11:37:39.907+0000 7fcd0d632700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2022-04-09T11:37:39.909 INFO:teuthology.orchestra.run.smithi119.stderr:[errno 2] RADOS object not found (error connecting to the cluster) 2022-04-09T11:37:39.910 INFO:teuthology.orchestra.run.smithi119.stderr:command terminated with exit code 1 2022-04-09T11:37:39.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-09T11:37:49.915 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch status -f json 2022-04-09T11:37:51.095 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:37:51.096 INFO:teuthology.orchestra.run.smithi119.stderr:no valid command found; 10 closest matches: 2022-04-09T11:37:51.096 INFO:teuthology.orchestra.run.smithi119.stderr:pg stat 2022-04-09T11:37:51.097 INFO:teuthology.orchestra.run.smithi119.stderr:pg getmap 2022-04-09T11:37:51.097 INFO:teuthology.orchestra.run.smithi119.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-04-09T11:37:51.097 INFO:teuthology.orchestra.run.smithi119.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-04-09T11:37:51.097 INFO:teuthology.orchestra.run.smithi119.stderr:pg dump_pools_json 2022-04-09T11:37:51.098 INFO:teuthology.orchestra.run.smithi119.stderr:pg ls-by-pool [...] 2022-04-09T11:37:51.098 INFO:teuthology.orchestra.run.smithi119.stderr:pg ls-by-primary [] [...] 2022-04-09T11:37:51.098 INFO:teuthology.orchestra.run.smithi119.stderr:pg ls-by-osd [] [...] 2022-04-09T11:37:51.098 INFO:teuthology.orchestra.run.smithi119.stderr:pg ls [] [...] 2022-04-09T11:37:51.099 INFO:teuthology.orchestra.run.smithi119.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-04-09T11:37:51.099 INFO:teuthology.orchestra.run.smithi119.stderr:Error EINVAL: invalid command 2022-04-09T11:37:51.100 INFO:teuthology.orchestra.run.smithi119.stderr:command terminated with exit code 22 2022-04-09T11:37:51.103 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-04-09T11:37:53.701 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:53.703832 I | op-mon: mons running: [a] 2022-04-09T11:37:53.921 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:53.924497 I | op-mon: Monitors in quorum: [a] 2022-04-09T11:37:53.922 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:53.924524 I | op-mon: mons created: 1 2022-04-09T11:37:54.131 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:54.132003 I | op-mon: waiting for mon quorum with [a] 2022-04-09T11:37:54.133 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:54.137459 I | op-mon: mons running: [a] 2022-04-09T11:37:54.345 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:54.348973 I | op-mon: Monitors in quorum: [a] 2022-04-09T11:37:54.346 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:54.349018 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-04-09T11:37:54.638 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:54.640949 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-04-09T11:37:54.638 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:54.640982 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-04-09T11:37:54.920 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:54.922992 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-04-09T11:37:54.920 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:54.923026 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-04-09T11:37:55.212 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:55.215237 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-04-09T11:37:55.212 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:55.215278 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-04-09T11:37:55.470 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:55.473243 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-04-09T11:37:55.470 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:55.473270 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-04-09T11:37:55.746 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:55.749440 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-04-09T11:37:55.747 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:55.749469 I | op-config: deleting "log file" option from the mon configuration database 2022-04-09T11:37:55.962 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:55.965317 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-04-09T11:37:55.963 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:55.965359 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-04-09T11:37:56.237 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:56.240934 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-04-09T11:37:56.530 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:56.533462 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-04-09T11:37:56.811 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:56.814064 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-04-09T11:37:57.134 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:57.137034 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-04-09T11:37:57.134 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:57.137071 I | cephclient: getting or creating ceph auth key "client.crash" 2022-04-09T11:37:57.480 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:57.483254 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-04-09T11:37:57.701 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:57.704425 I | cephclient: successfully enabled msgr2 protocol 2022-04-09T11:37:57.701 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:57.704450 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-04-09T11:37:57.702 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:57.704455 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-04-09T11:37:57.982 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:57.985029 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-04-09T11:37:58.000 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:58.003476 I | op-mgr: start running mgr 2022-04-09T11:37:58.000 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:37:58.003499 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-04-09T11:38:01.105 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch status -f json 2022-04-09T11:38:01.522 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:38:01.522 INFO:teuthology.orchestra.run.smithi119.stderr:no valid command found; 10 closest matches: 2022-04-09T11:38:01.522 INFO:teuthology.orchestra.run.smithi119.stderr:pg stat 2022-04-09T11:38:01.523 INFO:teuthology.orchestra.run.smithi119.stderr:pg getmap 2022-04-09T11:38:01.523 INFO:teuthology.orchestra.run.smithi119.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-04-09T11:38:01.523 INFO:teuthology.orchestra.run.smithi119.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-04-09T11:38:01.523 INFO:teuthology.orchestra.run.smithi119.stderr:pg dump_pools_json 2022-04-09T11:38:01.524 INFO:teuthology.orchestra.run.smithi119.stderr:pg ls-by-pool [...] 2022-04-09T11:38:01.524 INFO:teuthology.orchestra.run.smithi119.stderr:pg ls-by-primary [] [...] 2022-04-09T11:38:01.524 INFO:teuthology.orchestra.run.smithi119.stderr:pg ls-by-osd [] [...] 2022-04-09T11:38:01.524 INFO:teuthology.orchestra.run.smithi119.stderr:pg ls [] [...] 2022-04-09T11:38:01.524 INFO:teuthology.orchestra.run.smithi119.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-04-09T11:38:01.525 INFO:teuthology.orchestra.run.smithi119.stderr:Error EINVAL: invalid command 2022-04-09T11:38:01.525 INFO:teuthology.orchestra.run.smithi119.stderr:command terminated with exit code 22 2022-04-09T11:38:01.526 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-04-09T11:38:07.921 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:07.924870 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mgr-a" 2022-04-09T11:38:08.000 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:08.003517 I | op-mgr: setting services to point to mgr "a" 2022-04-09T11:38:08.305 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:08.308680 I | op-osd: start running osds in namespace "rook-ceph" 2022-04-09T11:38:08.308 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:08.309315 W | op-osd: useAllNodes is set to false and no nodes, storageClassDevicesets or volumeSources are specified, no OSD pods are going to be created 2022-04-09T11:38:08.308 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:08.309332 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-04-09T11:38:08.354 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:08.356829 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-04-09T11:38:08.461 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:08.464966 I | op-osd: no storageClassDeviceSets defined to configure OSDs on PVCs 2022-04-09T11:38:08.462 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:08.464990 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-04-09T11:38:08.462 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:08.464995 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-04-09T11:38:08.636 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:08.639915 W | cephclient: failed to enable mgr module "rook". trying again... 2022-04-09T11:38:08.873 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:08.876391 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-04-09T11:38:08.873 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:08.876433 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-04-09T11:38:08.879 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:08.882697 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph" 2022-04-09T11:38:08.879 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:08.882739 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "rook-ceph" 2022-04-09T11:38:08.879 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:08.882758 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph" 2022-04-09T11:38:08.912 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:08.915938 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-04-09T11:38:08.912 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:08.915964 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-04-09T11:38:08.915 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:08.917436 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Normal:ReconcileSucceeded:successfully configured CephCluster "rook-ceph/rook-ceph" 2022-04-09T11:38:08.915 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:08.917636 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-04-09T11:38:08.916 INFO:tasks.rook.operator.smithi119.stdout:I0409 11:38:08.917796 7 manager.go:118] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-04-09T11:38:08.940 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:08.944115 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:38:08.976 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:08.980058 I | op-mgr: successful modules: dashboard 2022-04-09T11:38:08.982 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:08.985390 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7... 2022-04-09T11:38:09.222 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:09.225990 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-04-09T11:38:10.421 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:10.425150 I | op-mgr: successful modules: prometheus 2022-04-09T11:38:11.527 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch status -f json 2022-04-09T11:38:11.943 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:38:11.943 INFO:teuthology.orchestra.run.smithi119.stderr:no valid command found; 10 closest matches: 2022-04-09T11:38:11.944 INFO:teuthology.orchestra.run.smithi119.stderr:pg stat 2022-04-09T11:38:11.944 INFO:teuthology.orchestra.run.smithi119.stderr:pg getmap 2022-04-09T11:38:11.944 INFO:teuthology.orchestra.run.smithi119.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-04-09T11:38:11.944 INFO:teuthology.orchestra.run.smithi119.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-04-09T11:38:11.945 INFO:teuthology.orchestra.run.smithi119.stderr:pg dump_pools_json 2022-04-09T11:38:11.945 INFO:teuthology.orchestra.run.smithi119.stderr:pg ls-by-pool [...] 2022-04-09T11:38:11.945 INFO:teuthology.orchestra.run.smithi119.stderr:pg ls-by-primary [] [...] 2022-04-09T11:38:11.945 INFO:teuthology.orchestra.run.smithi119.stderr:pg ls-by-osd [] [...] 2022-04-09T11:38:11.945 INFO:teuthology.orchestra.run.smithi119.stderr:pg ls [] [...] 2022-04-09T11:38:11.946 INFO:teuthology.orchestra.run.smithi119.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-04-09T11:38:11.946 INFO:teuthology.orchestra.run.smithi119.stderr:Error EINVAL: invalid command 2022-04-09T11:38:11.946 INFO:teuthology.orchestra.run.smithi119.stderr:command terminated with exit code 22 2022-04-09T11:38:11.947 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-04-09T11:38:14.755 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:14.759361 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-04-09T11:38:15.130 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:15.133754 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11488 quincy" 2022-04-09T11:38:15.130 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:15.133778 I | ceph-cluster-controller: validating ceph version from provided image 2022-04-09T11:38:15.131 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:15.133792 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11488 quincy", pursuing 2022-04-09T11:38:15.133 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:15.137146 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:38:15.518 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:15.521592 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-09T11:38:15.518 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:15.521743 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-09T11:38:15.663 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:15.667232 I | op-mgr: successful modules: mgr module(s) from the spec 2022-04-09T11:38:15.726 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:15.729905 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11488 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7" 2022-04-09T11:38:15.857 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:15.861304 I | op-mon: start running mons 2022-04-09T11:38:15.878 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:15.881894 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:38:15.891 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:15.895655 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.24.44:6789"]}] data:a=10.107.24.44:6789 mapping:{"node":{"a":{"Name":"smithi119","Hostname":"smithi119","Address":"172.21.15.119"}}} maxMonId:0] 2022-04-09T11:38:15.897 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:15.901464 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-09T11:38:15.897 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:15.901593 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-09T11:38:17.007 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:17.011275 I | op-mon: targeting the mon count 1 2022-04-09T11:38:17.010 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:17.014835 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-04-09T11:38:17.231 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:17.235220 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-04-09T11:38:17.232 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:17.235255 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-04-09T11:38:17.245 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:17.249357 I | cephclient: command failed for set rook backend. trying again... 2022-04-09T11:38:17.463 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:17.466875 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-04-09T11:38:17.463 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:17.466908 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-04-09T11:38:17.683 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:17.686327 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-04-09T11:38:17.683 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:17.686357 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-04-09T11:38:17.902 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:17.905729 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-04-09T11:38:17.902 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:17.905757 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-04-09T11:38:18.113 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:18.117394 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-04-09T11:38:18.114 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:18.117422 I | op-config: deleting "log file" option from the mon configuration database 2022-04-09T11:38:18.331 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:18.334771 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-04-09T11:38:18.331 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:18.334808 I | op-mon: checking for basic quorum with existing mons 2022-04-09T11:38:18.442 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:18.446434 I | op-mon: mon "a" endpoint is [v2:10.107.24.44:3300,v1:10.107.24.44:6789] 2022-04-09T11:38:18.460 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:18.463781 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.24.44:6789"]}] data:a=10.107.24.44:6789 mapping:{"node":{"a":{"Name":"smithi119","Hostname":"smithi119","Address":"172.21.15.119"}}} maxMonId:0] 2022-04-09T11:38:19.007 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:19.011064 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-09T11:38:19.010 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:19.011359 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-09T11:38:19.418 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:19.422112 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-04-09T11:38:19.424 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:19.428484 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-04-09T11:38:19.425 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:19.428504 I | op-mon: waiting for mon quorum with [a] 2022-04-09T11:38:19.624 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:19.628182 I | op-mon: mons running: [a] 2022-04-09T11:38:19.874 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:19.878266 I | op-mon: Monitors in quorum: [a] 2022-04-09T11:38:19.875 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:19.878292 I | op-mon: mons created: 1 2022-04-09T11:38:20.045 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:20.049166 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-04-09T11:38:20.116 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:20.120300 I | op-mon: waiting for mon quorum with [a] 2022-04-09T11:38:20.125 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:20.129034 I | op-mon: mons running: [a] 2022-04-09T11:38:20.376 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:20.379821 I | op-mon: Monitors in quorum: [a] 2022-04-09T11:38:20.376 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:20.379861 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-04-09T11:38:20.598 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:20.602308 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-04-09T11:38:20.823 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:20.826455 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-04-09T11:38:21.064 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:21.068499 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-04-09T11:38:21.407 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:21.411142 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-04-09T11:38:21.408 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:21.411180 I | cephclient: getting or creating ceph auth key "client.crash" 2022-04-09T11:38:21.808 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:21.812117 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-04-09T11:38:21.949 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch status -f json 2022-04-09T11:38:22.039 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:22.043631 I | cephclient: successfully enabled msgr2 protocol 2022-04-09T11:38:22.040 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:22.043669 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-04-09T11:38:22.040 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:22.043679 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-04-09T11:38:22.340 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:22.343785 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-04-09T11:38:22.358 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:22.362111 I | op-mgr: start running mgr 2022-04-09T11:38:22.358 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:22.362139 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-04-09T11:38:22.388 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:38:22.389 INFO:teuthology.orchestra.run.smithi119.stderr:no valid command found; 10 closest matches: 2022-04-09T11:38:22.389 INFO:teuthology.orchestra.run.smithi119.stderr:pg stat 2022-04-09T11:38:22.389 INFO:teuthology.orchestra.run.smithi119.stderr:pg getmap 2022-04-09T11:38:22.390 INFO:teuthology.orchestra.run.smithi119.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-04-09T11:38:22.390 INFO:teuthology.orchestra.run.smithi119.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-04-09T11:38:22.390 INFO:teuthology.orchestra.run.smithi119.stderr:pg dump_pools_json 2022-04-09T11:38:22.390 INFO:teuthology.orchestra.run.smithi119.stderr:pg ls-by-pool [...] 2022-04-09T11:38:22.391 INFO:teuthology.orchestra.run.smithi119.stderr:pg ls-by-primary [] [...] 2022-04-09T11:38:22.391 INFO:teuthology.orchestra.run.smithi119.stderr:pg ls-by-osd [] [...] 2022-04-09T11:38:22.391 INFO:teuthology.orchestra.run.smithi119.stderr:pg ls [] [...] 2022-04-09T11:38:22.391 INFO:teuthology.orchestra.run.smithi119.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-04-09T11:38:22.391 INFO:teuthology.orchestra.run.smithi119.stderr:Error EINVAL: invalid command 2022-04-09T11:38:22.393 INFO:teuthology.orchestra.run.smithi119.stderr:command terminated with exit code 22 2022-04-09T11:38:22.395 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-04-09T11:38:22.604 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:22.608719 I | cephclient: command failed for set rook backend. trying again... 2022-04-09T11:38:22.839 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:22.842898 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-04-09T11:38:22.847 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:22.851864 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-04-09T11:38:22.850 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:22.853710 I | op-mgr: setting services to point to mgr "a" 2022-04-09T11:38:23.621 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:23.624844 I | op-osd: start running osds in namespace "rook-ceph" 2022-04-09T11:38:23.621 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:23.624870 W | op-osd: useAllNodes is set to false and no nodes, storageClassDevicesets or volumeSources are specified, no OSD pods are going to be created 2022-04-09T11:38:23.622 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:23.624878 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-04-09T11:38:23.641 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:23.644752 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-04-09T11:38:23.807 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:23.811062 I | op-osd: no storageClassDeviceSets defined to configure OSDs on PVCs 2022-04-09T11:38:23.807 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:23.811095 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-04-09T11:38:23.808 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:23.811101 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-04-09T11:38:24.463 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:24.467576 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-04-09T11:38:24.464 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:24.467602 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-04-09T11:38:24.563 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:24.566213 I | op-mgr: successful modules: prometheus 2022-04-09T11:38:24.567 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:24.571314 I | op-mgr: successful modules: mgr module(s) from the spec 2022-04-09T11:38:24.571 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:24.572905 I | op-mgr: successful modules: dashboard 2022-04-09T11:38:24.794 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:24.797930 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-04-09T11:38:25.338 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:25.342054 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-04-09T11:38:25.915 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:25.918777 I | cephclient: command failed for set rook backend. trying again... 2022-04-09T11:38:27.916 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:27.920164 I | cephclient: command failed for set rook backend. trying again... 2022-04-09T11:38:30.080 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:30.084620 W | cephclient: failed to set mgr module mode "upmap". trying again... 2022-04-09T11:38:30.754 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:30.758155 I | op-mgr: successful modules: balancer 2022-04-09T11:38:31.379 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:31.383198 I | cephclient: action set rook backend succeeded on attempt 1 2022-04-09T11:38:31.379 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:31.383226 I | op-mgr: successful modules: orchestrator modules 2022-04-09T11:38:32.397 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch status -f json 2022-04-09T11:38:32.982 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:38:32.983 INFO:teuthology.orchestra.run.smithi119.stdout:{"available": true, "backend": "rook"} 2022-04-09T11:38:33.001 INFO:tasks.rook: mgr/rook orchestrator is active 2022-04-09T11:38:33.002 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph config set mgr mgr/rook/storage_class scratch 2022-04-09T11:38:33.287 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:33.291810 I | cephclient: action set rook backend succeeded on attempt 3 2022-04-09T11:38:33.288 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:33.291831 I | op-mgr: successful modules: orchestrator modules 2022-04-09T11:38:33.558 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch apply osd --all-available-devices 2022-04-09T11:38:34.199 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:34.203092 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-04-09T11:38:34.200 INFO:teuthology.orchestra.run.smithi119.stdout:['Success'] 2022-04-09T11:38:34.202 INFO:tasks.rook.operator.smithi119.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", AllowUnsupported: true}, 2022-04-09T11:38:34.202 INFO:tasks.rook.operator.smithi119.stdout: Storage: v1.StorageScopeSpec{ 2022-04-09T11:38:34.202 INFO:tasks.rook.operator.smithi119.stdout: ... // 3 identical fields 2022-04-09T11:38:34.202 INFO:tasks.rook.operator.smithi119.stdout: Config: nil, 2022-04-09T11:38:34.203 INFO:tasks.rook.operator.smithi119.stdout: Selection: {}, 2022-04-09T11:38:34.203 INFO:tasks.rook.operator.smithi119.stdout:- StorageClassDeviceSets: nil, 2022-04-09T11:38:34.203 INFO:tasks.rook.operator.smithi119.stdout:+ StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-04-09T11:38:34.203 INFO:tasks.rook.operator.smithi119.stdout:+ { 2022-04-09T11:38:34.203 INFO:tasks.rook.operator.smithi119.stdout:+ Name: "smithi119-lv-1", 2022-04-09T11:38:34.204 INFO:tasks.rook.operator.smithi119.stdout:+ Count: 1, 2022-04-09T11:38:34.204 INFO:tasks.rook.operator.smithi119.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-04-09T11:38:34.204 INFO:tasks.rook.operator.smithi119.stdout:+ }, 2022-04-09T11:38:34.206 INFO:tasks.rook.operator.smithi119.stdout:+ { 2022-04-09T11:38:34.207 INFO:tasks.rook.operator.smithi119.stdout:+ Name: "smithi119-lv-2", 2022-04-09T11:38:34.207 INFO:tasks.rook.operator.smithi119.stdout:+ Count: 1, 2022-04-09T11:38:34.207 INFO:tasks.rook.operator.smithi119.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-04-09T11:38:34.207 INFO:tasks.rook.operator.smithi119.stdout:+ }, 2022-04-09T11:38:34.207 INFO:tasks.rook.operator.smithi119.stdout:+ { 2022-04-09T11:38:34.208 INFO:tasks.rook.operator.smithi119.stdout:+ Name: "smithi119-lv-3", 2022-04-09T11:38:34.208 INFO:tasks.rook.operator.smithi119.stdout:+ Count: 1, 2022-04-09T11:38:34.208 INFO:tasks.rook.operator.smithi119.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-04-09T11:38:34.208 INFO:tasks.rook.operator.smithi119.stdout:+ }, 2022-04-09T11:38:34.209 INFO:tasks.rook.operator.smithi119.stdout:+ { 2022-04-09T11:38:34.209 INFO:tasks.rook.operator.smithi119.stdout:+ Name: "smithi119-lv-4", 2022-04-09T11:38:34.209 INFO:tasks.rook.operator.smithi119.stdout:+ Count: 1, 2022-04-09T11:38:34.209 INFO:tasks.rook.operator.smithi119.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-04-09T11:38:34.209 INFO:tasks.rook.operator.smithi119.stdout:+ }, 2022-04-09T11:38:34.210 INFO:tasks.rook.operator.smithi119.stdout:+ }, 2022-04-09T11:38:34.210 INFO:tasks.rook.operator.smithi119.stdout: }, 2022-04-09T11:38:34.210 INFO:tasks.rook.operator.smithi119.stdout: Annotations: nil, 2022-04-09T11:38:34.210 INFO:tasks.rook.operator.smithi119.stdout: Labels: nil, 2022-04-09T11:38:34.211 INFO:tasks.rook.operator.smithi119.stdout: ... // 20 identical fields 2022-04-09T11:38:34.211 INFO:tasks.rook.operator.smithi119.stdout: } 2022-04-09T11:38:34.211 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:34.203217 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-04-09T11:38:34.223 INFO:tasks.rook:Waiting for 4 OSDs 2022-04-09T11:38:34.224 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph osd stat -f json 2022-04-09T11:38:34.249 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:34.251233 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:38:34.258 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:34.262951 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7... 2022-04-09T11:38:34.709 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:38:34.727 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-04-09T11:38:34.728 INFO:tasks.rook: have 0/4 OSDs 2022-04-09T11:38:35.408 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:35.412753 I | op-mgr: successful modules: balancer 2022-04-09T11:38:39.915 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:39.918447 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11488 quincy" 2022-04-09T11:38:39.915 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:39.918478 I | ceph-cluster-controller: validating ceph version from provided image 2022-04-09T11:38:39.916 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:39.918492 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11488 quincy", pursuing 2022-04-09T11:38:39.918 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:39.923057 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:38:40.235 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:40.239762 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-09T11:38:40.236 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:40.240038 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-09T11:38:40.643 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:40.643980 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11488 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7" 2022-04-09T11:38:40.709 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:40.713278 I | op-mon: start running mons 2022-04-09T11:38:40.796 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:40.800108 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:38:41.015 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:41.019711 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.24.44:6789"]}] data:a=10.107.24.44:6789 mapping:{"node":{"a":{"Name":"smithi119","Hostname":"smithi119","Address":"172.21.15.119"}}} maxMonId:0] 2022-04-09T11:38:41.350 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:41.354506 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-09T11:38:41.350 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:41.354795 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-09T11:38:41.768 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:41.772652 I | op-mon: targeting the mon count 1 2022-04-09T11:38:41.782 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:41.785474 I | ceph-cluster-controller: Cluster update requested. CANCELLING CURRENT ORCHESTRATION 2022-04-09T11:38:42.782 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:42.786484 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-04-09T11:38:42.788 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:42.792904 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:38:42.803 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:42.808245 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7... 2022-04-09T11:38:42.834 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:42.838787 I | op-k8sutil: Removing previous job rook-ceph-detect-version to start a new one 2022-04-09T11:38:42.846 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:42.851052 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-04-09T11:38:44.729 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph osd stat -f json 2022-04-09T11:38:44.857 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:44.859606 I | op-k8sutil: batch job rook-ceph-detect-version deleted 2022-04-09T11:38:45.235 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:38:45.256 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-04-09T11:38:45.256 INFO:tasks.rook: have 0/4 OSDs 2022-04-09T11:38:50.952 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:50.955794 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11488 quincy" 2022-04-09T11:38:50.952 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:50.955821 I | ceph-cluster-controller: validating ceph version from provided image 2022-04-09T11:38:50.952 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:50.955835 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11488 quincy", pursuing 2022-04-09T11:38:51.217 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:51.221549 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:38:51.284 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:51.288310 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-09T11:38:51.287 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:51.288850 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-09T11:38:51.720 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:51.724229 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11488 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7" 2022-04-09T11:38:51.833 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:51.837275 I | op-mon: start running mons 2022-04-09T11:38:51.930 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:51.934430 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:38:52.431 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:52.435449 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.24.44:6789"]}] data:a=10.107.24.44:6789 mapping:{"node":{"a":{"Name":"smithi119","Hostname":"smithi119","Address":"172.21.15.119"}}} maxMonId:0] 2022-04-09T11:38:52.460 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:52.465219 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-09T11:38:52.463 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:52.465557 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-09T11:38:52.749 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:52.753747 I | op-mon: targeting the mon count 1 2022-04-09T11:38:52.754 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:52.758814 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-04-09T11:38:53.093 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:53.097084 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-04-09T11:38:53.093 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:53.097115 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-04-09T11:38:53.427 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:53.431812 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-04-09T11:38:53.428 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:53.431837 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-04-09T11:38:53.765 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:53.769404 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-04-09T11:38:53.765 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:53.769435 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-04-09T11:38:54.153 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:54.157842 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-04-09T11:38:54.154 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:54.157877 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-04-09T11:38:54.487 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:54.491553 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-04-09T11:38:54.487 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:54.491582 I | op-config: deleting "log file" option from the mon configuration database 2022-04-09T11:38:54.823 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:54.828222 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-04-09T11:38:54.824 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:54.828253 I | op-mon: checking for basic quorum with existing mons 2022-04-09T11:38:54.954 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:54.958887 I | op-mon: mon "a" endpoint is [v2:10.107.24.44:3300,v1:10.107.24.44:6789] 2022-04-09T11:38:54.975 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:54.980221 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.24.44:6789"]}] data:a=10.107.24.44:6789 mapping:{"node":{"a":{"Name":"smithi119","Hostname":"smithi119","Address":"172.21.15.119"}}} maxMonId:0] 2022-04-09T11:38:54.983 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:54.987816 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-09T11:38:54.985 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:54.988000 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-09T11:38:55.240 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:55.244694 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-04-09T11:38:55.258 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph osd stat -f json 2022-04-09T11:38:55.271 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:55.276455 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-04-09T11:38:55.272 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:55.276480 I | op-mon: waiting for mon quorum with [a] 2022-04-09T11:38:55.433 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:55.437626 I | op-mon: mons running: [a] 2022-04-09T11:38:55.788 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:38:55.807 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-04-09T11:38:55.807 INFO:tasks.rook: have 0/4 OSDs 2022-04-09T11:38:55.861 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:55.866389 I | op-mon: Monitors in quorum: [a] 2022-04-09T11:38:55.862 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:55.866407 I | op-mon: mons created: 1 2022-04-09T11:38:56.262 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:56.267266 I | op-mon: waiting for mon quorum with [a] 2022-04-09T11:38:56.269 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:56.274478 I | op-mon: mons running: [a] 2022-04-09T11:38:56.666 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:56.670793 I | op-mon: Monitors in quorum: [a] 2022-04-09T11:38:56.666 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:56.670839 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-04-09T11:38:57.082 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:57.087312 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-04-09T11:38:57.534 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:57.538708 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-04-09T11:38:57.946 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:57.950787 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-04-09T11:38:58.384 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:58.389288 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-04-09T11:38:58.385 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:58.389313 I | cephclient: getting or creating ceph auth key "client.crash" 2022-04-09T11:38:58.800 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:58.805473 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-04-09T11:38:59.194 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:59.198803 I | cephclient: successfully enabled msgr2 protocol 2022-04-09T11:38:59.194 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:59.198835 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-04-09T11:38:59.195 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:59.198845 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-04-09T11:38:59.623 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:59.627906 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-04-09T11:38:59.646 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:59.651097 I | op-mgr: start running mgr 2022-04-09T11:38:59.646 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:38:59.651122 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-04-09T11:39:00.107 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:00.111653 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-04-09T11:39:00.115 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:00.120838 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-04-09T11:39:00.120 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:00.126080 I | op-mgr: setting services to point to mgr "a" 2022-04-09T11:39:00.241 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:00.246262 I | op-osd: start running osds in namespace "rook-ceph" 2022-04-09T11:39:00.241 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:00.246294 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-04-09T11:39:00.265 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:00.270768 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-04-09T11:39:00.369 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:00.373916 I | op-osd: creating 1 new PVCs for device set "smithi119-lv-1" 2022-04-09T11:39:00.574 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:00.579232 I | op-osd: successfully provisioned PVC "smithi119-lv-1-data-0q96qb" 2022-04-09T11:39:00.577 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:00.579722 I | op-osd: creating 1 new PVCs for device set "smithi119-lv-2" 2022-04-09T11:39:00.791 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:00.795590 I | op-osd: successfully provisioned PVC "smithi119-lv-2-data-0d2cw5" 2022-04-09T11:39:00.791 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:00.795628 I | op-osd: creating 1 new PVCs for device set "smithi119-lv-3" 2022-04-09T11:39:00.972 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:00.976865 I | op-osd: successfully provisioned PVC "smithi119-lv-3-data-05666c" 2022-04-09T11:39:00.972 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:00.977013 I | op-osd: creating 1 new PVCs for device set "smithi119-lv-4" 2022-04-09T11:39:01.172 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:01.177202 I | op-osd: successfully provisioned PVC "smithi119-lv-4-data-0ts767" 2022-04-09T11:39:01.182 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:01.187555 I | op-osd: OSD will have its main bluestore block on "smithi119-lv-1-data-0q96qb" 2022-04-09T11:39:01.587 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:01.591705 I | op-osd: started OSD provisioning job for PVC "smithi119-lv-1-data-0q96qb" 2022-04-09T11:39:01.587 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:01.591734 I | op-osd: OSD will have its main bluestore block on "smithi119-lv-2-data-0d2cw5" 2022-04-09T11:39:01.593 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:01.598676 I | op-mgr: successful modules: dashboard 2022-04-09T11:39:01.600 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:01.605066 I | op-mgr: successful modules: mgr module(s) from the spec 2022-04-09T11:39:01.600 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:01.605223 I | op-mgr: successful modules: prometheus 2022-04-09T11:39:01.919 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:01.923945 I | op-mgr: successful modules: balancer 2022-04-09T11:39:02.655 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:02.660457 I | op-osd: started OSD provisioning job for PVC "smithi119-lv-2-data-0d2cw5" 2022-04-09T11:39:02.656 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:02.660496 I | op-osd: OSD will have its main bluestore block on "smithi119-lv-3-data-05666c" 2022-04-09T11:39:03.016 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:03.020776 I | op-osd: started OSD provisioning job for PVC "smithi119-lv-3-data-05666c" 2022-04-09T11:39:03.016 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:03.020801 I | op-osd: OSD will have its main bluestore block on "smithi119-lv-4-data-0ts767" 2022-04-09T11:39:03.413 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:03.416537 I | op-osd: started OSD provisioning job for PVC "smithi119-lv-4-data-0ts767" 2022-04-09T11:39:03.414 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:03.416627 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-04-09T11:39:03.414 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:03.416652 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-04-09T11:39:03.516 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:03.521494 I | op-osd: OSD orchestration status for PVC smithi119-lv-1-data-0q96qb is "starting" 2022-04-09T11:39:03.517 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:03.521522 I | op-osd: OSD orchestration status for PVC smithi119-lv-2-data-0d2cw5 is "starting" 2022-04-09T11:39:03.517 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:03.521534 I | op-osd: OSD orchestration status for PVC smithi119-lv-3-data-05666c is "starting" 2022-04-09T11:39:03.518 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:03.521544 I | op-osd: OSD orchestration status for PVC smithi119-lv-4-data-0ts767 is "starting" 2022-04-09T11:39:03.621 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:03.626750 I | op-mgr: successful modules: orchestrator modules 2022-04-09T11:39:05.809 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph osd stat -f json 2022-04-09T11:39:06.850 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:39:06.869 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":6,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-04-09T11:39:06.869 INFO:tasks.rook: have 0/4 OSDs 2022-04-09T11:39:16.796 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:16.801501 I | op-osd: OSD orchestration status for node smithi119-lv-1-data-0q96qb is "orchestrating" 2022-04-09T11:39:16.871 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph osd stat -f json 2022-04-09T11:39:17.637 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:17.641599 I | op-osd: OSD orchestration status for PVC smithi119-lv-1-data-0q96qb is "orchestrating" 2022-04-09T11:39:18.579 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:39:18.599 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":6,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-04-09T11:39:18.599 INFO:tasks.rook: have 0/4 OSDs 2022-04-09T11:39:23.062 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:23.068134 I | op-osd: OSD orchestration status for node smithi119-lv-2-data-0d2cw5 is "orchestrating" 2022-04-09T11:39:23.119 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:23.117879 I | op-osd: OSD orchestration status for node smithi119-lv-4-data-0ts767 is "orchestrating" 2022-04-09T11:39:23.871 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:23.876650 I | op-osd: OSD orchestration status for PVC smithi119-lv-4-data-0ts767 is "orchestrating" 2022-04-09T11:39:23.904 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:23.910350 I | op-osd: OSD orchestration status for PVC smithi119-lv-2-data-0d2cw5 is "orchestrating" 2022-04-09T11:39:24.789 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:24.795124 I | op-osd: OSD orchestration status for node smithi119-lv-3-data-05666c is "orchestrating" 2022-04-09T11:39:25.032 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:25.037690 I | op-osd: OSD orchestration status for PVC smithi119-lv-1-data-0q96qb is "completed" 2022-04-09T11:39:25.033 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:25.037716 I | op-osd: creating OSD 0 on PVC "smithi119-lv-1-data-0q96qb" 2022-04-09T11:39:25.033 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:25.037723 I | op-osd: OSD will have its main bluestore block on "smithi119-lv-1-data-0q96qb" 2022-04-09T11:39:25.757 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:25.762059 I | op-osd: OSD orchestration status for PVC smithi119-lv-3-data-05666c is "orchestrating" 2022-04-09T11:39:26.343 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:26.347090 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi119": the object has been modified; please apply your changes to the latest version and try again 2022-04-09T11:39:28.600 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph osd stat -f json 2022-04-09T11:39:29.292 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:39:29.310 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":10,"num_osds":4,"num_up_osds":0,"osd_up_since":0,"num_in_osds":4,"osd_in_since":1649504367,"num_remapped_pgs":0} 2022-04-09T11:39:29.311 INFO:tasks.rook: have 0/4 OSDs 2022-04-09T11:39:30.846 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:30.850560 I | op-osd: OSD orchestration status for PVC smithi119-lv-4-data-0ts767 is "completed" 2022-04-09T11:39:31.205 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:30.850615 I | op-osd: creating OSD 1 on PVC "smithi119-lv-4-data-0ts767" 2022-04-09T11:39:31.205 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:30.850631 I | op-osd: OSD will have its main bluestore block on "smithi119-lv-4-data-0ts767" 2022-04-09T11:39:32.431 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:32.437015 I | op-osd: OSD orchestration status for PVC smithi119-lv-2-data-0d2cw5 is "completed" 2022-04-09T11:39:32.432 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:32.437125 I | op-osd: creating OSD 2 on PVC "smithi119-lv-2-data-0d2cw5" 2022-04-09T11:39:32.432 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:32.437184 I | op-osd: OSD will have its main bluestore block on "smithi119-lv-2-data-0d2cw5" 2022-04-09T11:39:34.221 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:34.219440 I | op-osd: OSD orchestration status for PVC smithi119-lv-3-data-05666c is "completed" 2022-04-09T11:39:34.221 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:34.219472 I | op-osd: creating OSD 3 on PVC "smithi119-lv-3-data-05666c" 2022-04-09T11:39:34.222 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:34.219483 I | op-osd: OSD will have its main bluestore block on "smithi119-lv-3-data-05666c" 2022-04-09T11:39:37.222 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:37.228116 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-04-09T11:39:37.222 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:39:37.228140 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-04-09T11:39:39.312 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph osd stat -f json 2022-04-09T11:39:40.187 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:39:40.205 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":10,"num_osds":4,"num_up_osds":0,"osd_up_since":0,"num_in_osds":4,"osd_in_since":1649504367,"num_remapped_pgs":0} 2022-04-09T11:39:40.205 INFO:tasks.rook: have 0/4 OSDs 2022-04-09T11:39:50.207 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph osd stat -f json 2022-04-09T11:39:50.724 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:39:50.741 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":10,"num_osds":4,"num_up_osds":0,"osd_up_since":0,"num_in_osds":4,"osd_in_since":1649504367,"num_remapped_pgs":0} 2022-04-09T11:39:50.742 INFO:tasks.rook: have 0/4 OSDs 2022-04-09T11:40:00.743 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph osd stat -f json 2022-04-09T11:40:01.316 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:40:01.336 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":10,"num_osds":4,"num_up_osds":0,"osd_up_since":0,"num_in_osds":4,"osd_in_since":1649504367,"num_remapped_pgs":0} 2022-04-09T11:40:01.336 INFO:tasks.rook: have 0/4 OSDs 2022-04-09T11:40:11.337 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph osd stat -f json 2022-04-09T11:40:11.892 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:40:11.913 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":19,"num_osds":4,"num_up_osds":2,"osd_up_since":1649504409,"num_in_osds":4,"osd_in_since":1649504367,"num_remapped_pgs":0} 2022-04-09T11:40:11.914 INFO:tasks.rook: have 2/4 OSDs 2022-04-09T11:40:21.915 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph osd stat -f json 2022-04-09T11:40:22.430 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:40:22.450 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":23,"num_osds":4,"num_up_osds":4,"osd_up_since":1649504412,"num_in_osds":4,"osd_in_since":1649504367,"num_remapped_pgs":0} 2022-04-09T11:40:22.450 INFO:tasks.rook:Distributing ceph config and client.admin keyring 2022-04-09T11:40:22.451 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- cat /etc/ceph/ceph.conf 2022-04-09T11:40:22.623 INFO:teuthology.orchestra.run.smithi119.stdout:[global] 2022-04-09T11:40:22.623 INFO:teuthology.orchestra.run.smithi119.stdout:mon_host = 10.107.24.44:6789 2022-04-09T11:40:22.623 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:40:22.624 INFO:teuthology.orchestra.run.smithi119.stdout:[client.admin] 2022-04-09T11:40:22.624 INFO:teuthology.orchestra.run.smithi119.stdout:keyring = /etc/ceph/keyring 2022-04-09T11:40:22.627 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- cat /etc/ceph/keyring 2022-04-09T11:40:22.772 INFO:teuthology.orchestra.run.smithi119.stdout:[client.admin] 2022-04-09T11:40:22.773 INFO:teuthology.orchestra.run.smithi119.stdout:key = AQDgb1FiBNz0IRAAofkQaGcum9nbU74PrKripQ== 2022-04-09T11:40:22.777 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /etc/ceph 2022-04-09T11:40:22.789 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-04-09T11:40:22.789 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/ceph/ceph.conf 2022-04-09T11:40:22.804 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-04-09T11:40:22.805 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/ceph/keyring 2022-04-09T11:40:22.815 INFO:tasks.rook:Setting up client nodes... 2022-04-09T11:40:22.816 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph auth get-or-create client.a mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-04-09T11:40:23.460 INFO:teuthology.orchestra.run.smithi119.stdout:[client.a] 2022-04-09T11:40:23.460 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQCncFFizabOFxAA9x6U7VRhyA1fMCzXeFCY1Q== 2022-04-09T11:40:23.482 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-04-09T11:40:23.483 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/ceph/ceph.client.a.keyring 2022-04-09T11:40:23.483 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0644 /etc/ceph/ceph.client.a.keyring 2022-04-09T11:40:23.505 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph osd dump --format=json 2022-04-09T11:40:24.096 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:40:24.097 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":23,"fsid":"e4e3495d-f0bd-4ff6-adc5-034fdb2c4540","created":"2022-04-09T11:37:49.814129+0000","modified":"2022-04-09T11:40:16.078606+0000","last_up_change":"2022-04-09T11:40:12.861232+0000","last_in_change":"2022-04-09T11:39:27.532173+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-04-09T11:40:13.685217+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"2716106c-4b0d-46f2-a8ca-2395f644890e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.22:6800","nonce":494818532},{"type":"v1","addr":"10.251.176.22:6801","nonce":494818532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.22:6802","nonce":494818532},{"type":"v1","addr":"10.251.176.22:6803","nonce":494818532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.22:6806","nonce":494818532},{"type":"v1","addr":"10.251.176.22:6807","nonce":494818532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.22:6804","nonce":494818532},{"type":"v1","addr":"10.251.176.22:6805","nonce":494818532}]},"public_addr":"10.251.176.22:6801/494818532","cluster_addr":"10.251.176.22:6803/494818532","heartbeat_back_addr":"10.251.176.22:6807/494818532","heartbeat_front_addr":"10.251.176.22:6805/494818532","state":["exists","up"]},{"osd":1,"uuid":"e336dcca-c4d0-471d-95b2-7bca6dc0ae08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.25:6800","nonce":1616097300},{"type":"v1","addr":"10.251.176.25:6801","nonce":1616097300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.25:6802","nonce":1616097300},{"type":"v1","addr":"10.251.176.25:6803","nonce":1616097300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.25:6806","nonce":1616097300},{"type":"v1","addr":"10.251.176.25:6807","nonce":1616097300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.25:6804","nonce":1616097300},{"type":"v1","addr":"10.251.176.25:6805","nonce":1616097300}]},"public_addr":"10.251.176.25:6801/1616097300","cluster_addr":"10.251.176.25:6803/1616097300","heartbeat_back_addr":"10.251.176.25:6807/1616097300","heartbeat_front_addr":"10.251.176.25:6805/1616097300","state":["exists","up"]},{"osd":2,"uuid":"686f4abe-90b7-49cf-8469-da2a62e69ac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.24:6800","nonce":1749158363},{"type":"v1","addr":"10.251.176.24:6801","nonce":1749158363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.24:6802","nonce":1749158363},{"type":"v1","addr":"10.251.176.24:6803","nonce":1749158363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.24:6806","nonce":1749158363},{"type":"v1","addr":"10.251.176.24:6807","nonce":1749158363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.24:6804","nonce":1749158363},{"type":"v1","addr":"10.251.176.24:6805","nonce":1749158363}]},"public_addr":"10.251.176.24:6801/1749158363","cluster_addr":"10.251.176.24:6803/1749158363","heartbeat_back_addr":"10.251.176.24:6807/1749158363","heartbeat_front_addr":"10.251.176.24:6805/1749158363","state":["exists","up"]},{"osd":3,"uuid":"a76fc4ff-87f0-4bb8-b4c4-e5ef395e92bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.26:6800","nonce":3197094910},{"type":"v1","addr":"10.251.176.26:6801","nonce":3197094910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.26:6802","nonce":3197094910},{"type":"v1","addr":"10.251.176.26:6803","nonce":3197094910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.26:6806","nonce":3197094910},{"type":"v1","addr":"10.251.176.26:6807","nonce":3197094910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.26:6804","nonce":3197094910},{"type":"v1","addr":"10.251.176.26:6805","nonce":3197094910}]},"public_addr":"10.251.176.26:6801/3197094910","cluster_addr":"10.251.176.26:6803/3197094910","heartbeat_back_addr":"10.251.176.26:6807/3197094910","heartbeat_front_addr":"10.251.176.26:6805/3197094910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-09T11:40:01.500869+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-09T11:40:11.144288+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-09T11:40:07.865187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-09T11:40:11.137162+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-09T11:40:24.116 INFO:tasks.rook.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-04-09T11:40:13.685217+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'on', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-04-09T11:40:24.116 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph osd pool get .mgr pg_num 2022-04-09T11:40:24.624 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2022-04-09T11:40:24.646 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-04-09T11:40:24.647 INFO:tasks.rook.ceph_manager.ceph:waiting for mgr available 2022-04-09T11:40:24.647 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph mgr dump --format=json 2022-04-09T11:40:25.201 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:40:25.223 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":15,"active_gid":4198,"active_name":"a","active_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.13:6800","nonce":3727899180},{"type":"v1","addr":"10.251.176.13:6801","nonce":3727899180}]},"active_addr":"10.251.176.13:6801/3727899180","active_change":"2022-04-09T11:38:29.029499+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","prometheus","restful","rook"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"prometheus":"http://10.251.176.13:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":5,"active_clients":[{"addrvec":[{"type":"v2","addr":"10.251.176.13:0","nonce":5682893}]},{"addrvec":[{"type":"v2","addr":"10.251.176.13:0","nonce":1563447241}]},{"addrvec":[{"type":"v2","addr":"10.251.176.13:0","nonce":3043088856}]},{"addrvec":[{"type":"v2","addr":"10.251.176.13:0","nonce":2794144652}]}]}} 2022-04-09T11:40:25.227 INFO:tasks.rook.ceph_manager.ceph:mgr available! 2022-04-09T11:40:25.228 INFO:tasks.rook.ceph_manager.ceph:waiting for all up 2022-04-09T11:40:25.228 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph osd dump --format=json 2022-04-09T11:40:25.734 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:40:25.734 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":23,"fsid":"e4e3495d-f0bd-4ff6-adc5-034fdb2c4540","created":"2022-04-09T11:37:49.814129+0000","modified":"2022-04-09T11:40:16.078606+0000","last_up_change":"2022-04-09T11:40:12.861232+0000","last_in_change":"2022-04-09T11:39:27.532173+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-04-09T11:40:13.685217+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"2716106c-4b0d-46f2-a8ca-2395f644890e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.22:6800","nonce":494818532},{"type":"v1","addr":"10.251.176.22:6801","nonce":494818532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.22:6802","nonce":494818532},{"type":"v1","addr":"10.251.176.22:6803","nonce":494818532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.22:6806","nonce":494818532},{"type":"v1","addr":"10.251.176.22:6807","nonce":494818532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.22:6804","nonce":494818532},{"type":"v1","addr":"10.251.176.22:6805","nonce":494818532}]},"public_addr":"10.251.176.22:6801/494818532","cluster_addr":"10.251.176.22:6803/494818532","heartbeat_back_addr":"10.251.176.22:6807/494818532","heartbeat_front_addr":"10.251.176.22:6805/494818532","state":["exists","up"]},{"osd":1,"uuid":"e336dcca-c4d0-471d-95b2-7bca6dc0ae08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.25:6800","nonce":1616097300},{"type":"v1","addr":"10.251.176.25:6801","nonce":1616097300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.25:6802","nonce":1616097300},{"type":"v1","addr":"10.251.176.25:6803","nonce":1616097300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.25:6806","nonce":1616097300},{"type":"v1","addr":"10.251.176.25:6807","nonce":1616097300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.25:6804","nonce":1616097300},{"type":"v1","addr":"10.251.176.25:6805","nonce":1616097300}]},"public_addr":"10.251.176.25:6801/1616097300","cluster_addr":"10.251.176.25:6803/1616097300","heartbeat_back_addr":"10.251.176.25:6807/1616097300","heartbeat_front_addr":"10.251.176.25:6805/1616097300","state":["exists","up"]},{"osd":2,"uuid":"686f4abe-90b7-49cf-8469-da2a62e69ac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.24:6800","nonce":1749158363},{"type":"v1","addr":"10.251.176.24:6801","nonce":1749158363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.24:6802","nonce":1749158363},{"type":"v1","addr":"10.251.176.24:6803","nonce":1749158363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.24:6806","nonce":1749158363},{"type":"v1","addr":"10.251.176.24:6807","nonce":1749158363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.24:6804","nonce":1749158363},{"type":"v1","addr":"10.251.176.24:6805","nonce":1749158363}]},"public_addr":"10.251.176.24:6801/1749158363","cluster_addr":"10.251.176.24:6803/1749158363","heartbeat_back_addr":"10.251.176.24:6807/1749158363","heartbeat_front_addr":"10.251.176.24:6805/1749158363","state":["exists","up"]},{"osd":3,"uuid":"a76fc4ff-87f0-4bb8-b4c4-e5ef395e92bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.26:6800","nonce":3197094910},{"type":"v1","addr":"10.251.176.26:6801","nonce":3197094910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.26:6802","nonce":3197094910},{"type":"v1","addr":"10.251.176.26:6803","nonce":3197094910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.26:6806","nonce":3197094910},{"type":"v1","addr":"10.251.176.26:6807","nonce":3197094910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.26:6804","nonce":3197094910},{"type":"v1","addr":"10.251.176.26:6805","nonce":3197094910}]},"public_addr":"10.251.176.26:6801/3197094910","cluster_addr":"10.251.176.26:6803/3197094910","heartbeat_back_addr":"10.251.176.26:6807/3197094910","heartbeat_front_addr":"10.251.176.26:6805/3197094910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-09T11:40:01.500869+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-09T11:40:11.144288+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-09T11:40:07.865187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-09T11:40:11.137162+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-09T11:40:25.752 INFO:tasks.rook.ceph_manager.ceph:all up! 2022-04-09T11:40:25.752 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph osd dump --format=json 2022-04-09T11:40:26.244 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:40:26.245 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":23,"fsid":"e4e3495d-f0bd-4ff6-adc5-034fdb2c4540","created":"2022-04-09T11:37:49.814129+0000","modified":"2022-04-09T11:40:16.078606+0000","last_up_change":"2022-04-09T11:40:12.861232+0000","last_in_change":"2022-04-09T11:39:27.532173+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-04-09T11:40:13.685217+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"2716106c-4b0d-46f2-a8ca-2395f644890e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.22:6800","nonce":494818532},{"type":"v1","addr":"10.251.176.22:6801","nonce":494818532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.22:6802","nonce":494818532},{"type":"v1","addr":"10.251.176.22:6803","nonce":494818532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.22:6806","nonce":494818532},{"type":"v1","addr":"10.251.176.22:6807","nonce":494818532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.22:6804","nonce":494818532},{"type":"v1","addr":"10.251.176.22:6805","nonce":494818532}]},"public_addr":"10.251.176.22:6801/494818532","cluster_addr":"10.251.176.22:6803/494818532","heartbeat_back_addr":"10.251.176.22:6807/494818532","heartbeat_front_addr":"10.251.176.22:6805/494818532","state":["exists","up"]},{"osd":1,"uuid":"e336dcca-c4d0-471d-95b2-7bca6dc0ae08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.25:6800","nonce":1616097300},{"type":"v1","addr":"10.251.176.25:6801","nonce":1616097300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.25:6802","nonce":1616097300},{"type":"v1","addr":"10.251.176.25:6803","nonce":1616097300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.25:6806","nonce":1616097300},{"type":"v1","addr":"10.251.176.25:6807","nonce":1616097300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.25:6804","nonce":1616097300},{"type":"v1","addr":"10.251.176.25:6805","nonce":1616097300}]},"public_addr":"10.251.176.25:6801/1616097300","cluster_addr":"10.251.176.25:6803/1616097300","heartbeat_back_addr":"10.251.176.25:6807/1616097300","heartbeat_front_addr":"10.251.176.25:6805/1616097300","state":["exists","up"]},{"osd":2,"uuid":"686f4abe-90b7-49cf-8469-da2a62e69ac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.24:6800","nonce":1749158363},{"type":"v1","addr":"10.251.176.24:6801","nonce":1749158363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.24:6802","nonce":1749158363},{"type":"v1","addr":"10.251.176.24:6803","nonce":1749158363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.24:6806","nonce":1749158363},{"type":"v1","addr":"10.251.176.24:6807","nonce":1749158363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.24:6804","nonce":1749158363},{"type":"v1","addr":"10.251.176.24:6805","nonce":1749158363}]},"public_addr":"10.251.176.24:6801/1749158363","cluster_addr":"10.251.176.24:6803/1749158363","heartbeat_back_addr":"10.251.176.24:6807/1749158363","heartbeat_front_addr":"10.251.176.24:6805/1749158363","state":["exists","up"]},{"osd":3,"uuid":"a76fc4ff-87f0-4bb8-b4c4-e5ef395e92bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.26:6800","nonce":3197094910},{"type":"v1","addr":"10.251.176.26:6801","nonce":3197094910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.26:6802","nonce":3197094910},{"type":"v1","addr":"10.251.176.26:6803","nonce":3197094910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.26:6806","nonce":3197094910},{"type":"v1","addr":"10.251.176.26:6807","nonce":3197094910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.251.176.26:6804","nonce":3197094910},{"type":"v1","addr":"10.251.176.26:6805","nonce":3197094910}]},"public_addr":"10.251.176.26:6801/3197094910","cluster_addr":"10.251.176.26:6803/3197094910","heartbeat_back_addr":"10.251.176.26:6807/3197094910","heartbeat_front_addr":"10.251.176.26:6805/3197094910","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-09T11:40:01.500869+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-09T11:40:11.144288+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-09T11:40:07.865187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-09T11:40:11.137162+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-09T11:40:26.264 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph tell osd.0 flush_pg_stats 2022-04-09T11:40:26.264 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph tell osd.1 flush_pg_stats 2022-04-09T11:40:26.265 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph tell osd.2 flush_pg_stats 2022-04-09T11:40:26.266 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph tell osd.3 flush_pg_stats 2022-04-09T11:40:26.629 INFO:teuthology.orchestra.run.smithi119.stdout:85899345925 2022-04-09T11:40:26.630 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph osd last-stat-seq osd.1 2022-04-09T11:40:26.661 INFO:teuthology.orchestra.run.smithi119.stdout:73014444038 2022-04-09T11:40:26.662 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph osd last-stat-seq osd.2 2022-04-09T11:40:26.731 INFO:teuthology.orchestra.run.smithi119.stdout:85899345925 2022-04-09T11:40:26.732 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph osd last-stat-seq osd.3 2022-04-09T11:40:26.824 INFO:teuthology.orchestra.run.smithi119.stdout:55834574854 2022-04-09T11:40:26.825 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph osd last-stat-seq osd.0 2022-04-09T11:40:27.160 INFO:teuthology.orchestra.run.smithi119.stdout:85899345924 2022-04-09T11:40:27.181 INFO:tasks.rook.ceph_manager.ceph:need seq 85899345925 got 85899345924 for osd.1 2022-04-09T11:40:27.255 INFO:teuthology.orchestra.run.smithi119.stdout:85899345924 2022-04-09T11:40:27.269 INFO:teuthology.orchestra.run.smithi119.stdout:73014444036 2022-04-09T11:40:27.276 INFO:tasks.rook.ceph_manager.ceph:need seq 85899345925 got 85899345924 for osd.3 2022-04-09T11:40:27.291 INFO:tasks.rook.ceph_manager.ceph:need seq 73014444038 got 73014444036 for osd.2 2022-04-09T11:40:27.369 INFO:teuthology.orchestra.run.smithi119.stdout:55834574852 2022-04-09T11:40:27.392 INFO:tasks.rook.ceph_manager.ceph:need seq 55834574854 got 55834574852 for osd.0 2022-04-09T11:40:28.183 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph osd last-stat-seq osd.1 2022-04-09T11:40:28.278 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph osd last-stat-seq osd.3 2022-04-09T11:40:28.292 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph osd last-stat-seq osd.2 2022-04-09T11:40:28.393 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph osd last-stat-seq osd.0 2022-04-09T11:40:28.726 INFO:teuthology.orchestra.run.smithi119.stdout:85899345925 2022-04-09T11:40:28.744 INFO:tasks.rook.ceph_manager.ceph:need seq 85899345925 got 85899345925 for osd.1 2022-04-09T11:40:28.744 DEBUG:teuthology.parallel:result is None 2022-04-09T11:40:28.844 INFO:teuthology.orchestra.run.smithi119.stdout:85899345925 2022-04-09T11:40:28.863 INFO:tasks.rook.ceph_manager.ceph:need seq 85899345925 got 85899345925 for osd.3 2022-04-09T11:40:28.864 DEBUG:teuthology.parallel:result is None 2022-04-09T11:40:28.945 INFO:teuthology.orchestra.run.smithi119.stdout:73014444038 2022-04-09T11:40:28.964 INFO:tasks.rook.ceph_manager.ceph:need seq 73014444038 got 73014444038 for osd.2 2022-04-09T11:40:28.964 DEBUG:teuthology.parallel:result is None 2022-04-09T11:40:28.965 INFO:teuthology.orchestra.run.smithi119.stdout:55834574854 2022-04-09T11:40:28.982 INFO:tasks.rook.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.0 2022-04-09T11:40:28.982 DEBUG:teuthology.parallel:result is None 2022-04-09T11:40:28.983 INFO:tasks.rook.ceph_manager.ceph:waiting for clean 2022-04-09T11:40:28.983 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph pg dump --format=json 2022-04-09T11:40:29.477 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:40:29.480 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2022-04-09T11:40:29.495 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":71,"stamp":"2022-04-09T11:40:29.213995+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":4,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":374980608,"kb_used":23556,"kb_used_data":2148,"kb_used_omap":0,"kb_used_meta":20736,"kb_avail":374957052,"statfs":{"total":383980142592,"available":383956021248,"internally_reserved":0,"allocated":2199552,"data_stored":1589028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21233664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002343"},"pg_stats":[{"pgid":"1.0","version":"23'97","reported_seq":137,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-09T11:40:19.367060+0000","last_change":"2022-04-09T11:40:15.451292+0000","last_active":"2022-04-09T11:40:19.367060+0000","last_peered":"2022-04-09T11:40:19.367060+0000","last_clean":"2022-04-09T11:40:19.367060+0000","last_became_active":"2022-04-09T11:40:15.448378+0000","last_became_peered":"2022-04-09T11:40:15.448378+0000","last_unstale":"2022-04-09T11:40:19.367060+0000","last_undegraded":"2022-04-09T11:40:19.367060+0000","last_fullsized":"2022-04-09T11:40:19.367060+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-09T11:40:13.914598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-09T11:40:13.914598+0000","last_clean_scrub_stamp":"2022-04-09T11:40:13.914598+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-10T18:07:20.398343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1413120,"data_stored":1400868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":20,"seq":85899345926,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5064,"kb_used_data":192,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93740088,"statfs":{"total":95995035648,"available":95989850112,"internally_reserved":0,"allocated":196608,"data_stored":47040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83799999999999997}]}]},{"osd":1,"up_from":20,"seq":85899345926,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6036,"kb_used_data":652,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93739116,"statfs":{"total":95995035648,"available":95988854784,"internally_reserved":0,"allocated":667648,"data_stored":513996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.085}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6356,"kb_used_data":652,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738796,"statfs":{"total":95995035648,"available":95988527104,"internally_reserved":0,"allocated":667648,"data_stored":513996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]}]},{"osd":2,"up_from":17,"seq":73014444038,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6100,"kb_used_data":652,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93739052,"statfs":{"total":95995035648,"available":95988789248,"internally_reserved":0,"allocated":667648,"data_stored":513996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-04-09T11:40:29.496 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph pg dump --format=json 2022-04-09T11:40:30.007 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:40:30.010 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2022-04-09T11:40:30.026 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":71,"stamp":"2022-04-09T11:40:29.213995+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":4,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":374980608,"kb_used":23556,"kb_used_data":2148,"kb_used_omap":0,"kb_used_meta":20736,"kb_avail":374957052,"statfs":{"total":383980142592,"available":383956021248,"internally_reserved":0,"allocated":2199552,"data_stored":1589028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21233664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002343"},"pg_stats":[{"pgid":"1.0","version":"23'97","reported_seq":137,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-09T11:40:19.367060+0000","last_change":"2022-04-09T11:40:15.451292+0000","last_active":"2022-04-09T11:40:19.367060+0000","last_peered":"2022-04-09T11:40:19.367060+0000","last_clean":"2022-04-09T11:40:19.367060+0000","last_became_active":"2022-04-09T11:40:15.448378+0000","last_became_peered":"2022-04-09T11:40:15.448378+0000","last_unstale":"2022-04-09T11:40:19.367060+0000","last_undegraded":"2022-04-09T11:40:19.367060+0000","last_fullsized":"2022-04-09T11:40:19.367060+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-09T11:40:13.914598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-09T11:40:13.914598+0000","last_clean_scrub_stamp":"2022-04-09T11:40:13.914598+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-10T18:07:20.398343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1413120,"data_stored":1400868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":20,"seq":85899345926,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5064,"kb_used_data":192,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93740088,"statfs":{"total":95995035648,"available":95989850112,"internally_reserved":0,"allocated":196608,"data_stored":47040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83799999999999997}]}]},{"osd":1,"up_from":20,"seq":85899345926,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6036,"kb_used_data":652,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93739116,"statfs":{"total":95995035648,"available":95988854784,"internally_reserved":0,"allocated":667648,"data_stored":513996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.085}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6356,"kb_used_data":652,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738796,"statfs":{"total":95995035648,"available":95988527104,"internally_reserved":0,"allocated":667648,"data_stored":513996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]}]},{"osd":2,"up_from":17,"seq":73014444038,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6100,"kb_used_data":652,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93739052,"statfs":{"total":95995035648,"available":95988789248,"internally_reserved":0,"allocated":667648,"data_stored":513996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-04-09T11:40:30.027 INFO:tasks.rook.ceph_manager.ceph:clean! 2022-04-09T11:40:30.027 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-04-09T11:40:30.027 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy 2022-04-09T11:40:30.028 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph health --format=json 2022-04-09T11:40:30.555 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:40:30.556 INFO:teuthology.orchestra.run.smithi119.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-04-09T11:40:30.575 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy done 2022-04-09T11:40:30.576 INFO:tasks.rook:Rook complete, yielding 2022-04-09T11:40:30.576 INFO:teuthology.run_tasks:Running task rook.shell... 2022-04-09T11:40:30.589 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph -s 2022-04-09T11:40:31.143 INFO:teuthology.orchestra.run.smithi119.stdout: cluster: 2022-04-09T11:40:31.144 INFO:teuthology.orchestra.run.smithi119.stdout: id: e4e3495d-f0bd-4ff6-adc5-034fdb2c4540 2022-04-09T11:40:31.144 INFO:teuthology.orchestra.run.smithi119.stdout: health: HEALTH_OK 2022-04-09T11:40:31.144 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:40:31.144 INFO:teuthology.orchestra.run.smithi119.stdout: services: 2022-04-09T11:40:31.145 INFO:teuthology.orchestra.run.smithi119.stdout: mon: 1 daemons, quorum a (age 2m) 2022-04-09T11:40:31.145 INFO:teuthology.orchestra.run.smithi119.stdout: mgr: a(active, since 2m) 2022-04-09T11:40:31.145 INFO:teuthology.orchestra.run.smithi119.stdout: osd: 4 osds: 4 up (since 18s), 4 in (since 63s) 2022-04-09T11:40:31.145 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:40:31.145 INFO:teuthology.orchestra.run.smithi119.stdout: data: 2022-04-09T11:40:31.146 INFO:teuthology.orchestra.run.smithi119.stdout: pools: 1 pools, 1 pgs 2022-04-09T11:40:31.146 INFO:teuthology.orchestra.run.smithi119.stdout: objects: 2 objects, 577 KiB 2022-04-09T11:40:31.146 INFO:teuthology.orchestra.run.smithi119.stdout: usage: 23 MiB used, 358 GiB / 358 GiB avail 2022-04-09T11:40:31.146 INFO:teuthology.orchestra.run.smithi119.stdout: pgs: 1 active+clean 2022-04-09T11:40:31.147 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:40:31.147 INFO:teuthology.orchestra.run.smithi119.stdout: io: 2022-04-09T11:40:31.147 INFO:teuthology.orchestra.run.smithi119.stdout: client: 921 B/s rd, 26 KiB/s wr, 0 op/s rd, 1 op/s wr 2022-04-09T11:40:31.147 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:40:31.161 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch status 2022-04-09T11:40:31.810 INFO:teuthology.orchestra.run.smithi119.stdout:Backend: rook 2022-04-09T11:40:31.811 INFO:teuthology.orchestra.run.smithi119.stdout:Available: Yes 2022-04-09T11:40:31.833 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ps 2022-04-09T11:40:32.332 INFO:teuthology.orchestra.run.smithi119.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-04-09T11:40:32.333 INFO:teuthology.orchestra.run.smithi119.stdout:crashcollector.crash smithi119 running (66s) 0s ago 66s - - 2abf022dd4de 2022-04-09T11:40:32.333 INFO:teuthology.orchestra.run.smithi119.stdout:mgr.a smithi119 running (2m) 0s ago 2m - - 2abf022dd4de 2022-04-09T11:40:32.333 INFO:teuthology.orchestra.run.smithi119.stdout:mon.a smithi119 running (3m) 0s ago 3m - - 2abf022dd4de 2022-04-09T11:40:32.333 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0 smithi119 running (67s) 0s ago 67s - - 2abf022dd4de 2022-04-09T11:40:32.334 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1 smithi119 running (60s) 0s ago 60s - - 2abf022dd4de 2022-04-09T11:40:32.334 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2 smithi119 running (59s) 0s ago 60s - - 2abf022dd4de 2022-04-09T11:40:32.334 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3 smithi119 running (56s) 0s ago 57s - - 2abf022dd4de 2022-04-09T11:40:32.356 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls 2022-04-09T11:40:32.918 INFO:teuthology.orchestra.run.smithi119.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-04-09T11:40:32.918 INFO:teuthology.orchestra.run.smithi119.stdout:crash 1/1 0s ago 66s * 2022-04-09T11:40:32.919 INFO:teuthology.orchestra.run.smithi119.stdout:mgr 1/1 0s ago 2m count:1 2022-04-09T11:40:32.919 INFO:teuthology.orchestra.run.smithi119.stdout:mon 1/1 0s ago 3m count:1 2022-04-09T11:40:32.919 INFO:teuthology.orchestra.run.smithi119.stdout:osd 8 0s ago 67s 2022-04-09T11:40:32.919 INFO:teuthology.orchestra.run.smithi119.stdout:osd.all-available-devices 0 0s ago - * 2022-04-09T11:40:32.937 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch device ls 2022-04-09T11:40:33.472 INFO:teuthology.orchestra.run.smithi119.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-04-09T11:40:33.472 INFO:teuthology.orchestra.run.smithi119.stdout:smithi119 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-04-09T11:40:33.472 INFO:teuthology.orchestra.run.smithi119.stdout:smithi119 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-04-09T11:40:33.473 INFO:teuthology.orchestra.run.smithi119.stdout:smithi119 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-04-09T11:40:33.473 INFO:teuthology.orchestra.run.smithi119.stdout:smithi119 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-04-09T11:40:33.493 INFO:teuthology.run_tasks:Running task install... 2022-04-09T11:40:33.503 DEBUG:teuthology.task.install:project ceph 2022-04-09T11:40:33.503 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': 'a70e525ea98f7194c5e516757c6105ca14c81cb7'}} 2022-04-09T11:40:33.504 DEBUG:teuthology.task.install:config {'host.a': None, 'sha1': 'a70e525ea98f7194c5e516757c6105ca14c81cb7'} 2022-04-09T11:40:33.504 INFO:teuthology.task.install:Using flavor: default 2022-04-09T11:40:33.511 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2022-04-09T11:40:33.512 INFO:teuthology.task.install:extra packages: [] 2022-04-09T11:40:33.512 DEBUG:teuthology.orchestra.run.smithi119:> sudo lsb_release -is 2022-04-09T11:40:33.557 INFO:teuthology.orchestra.run.smithi119.stdout:Ubuntu 2022-04-09T11:40:33.563 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-04-09T11:40:33.564 DEBUG:teuthology.orchestra.run.smithi119:> sudo apt-key list | grep Ceph 2022-04-09T11:40:33.704 INFO:teuthology.orchestra.run.smithi119.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2022-04-09T11:40:33.778 INFO:teuthology.orchestra.run.smithi119.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2022-04-09T11:40:33.779 INFO:teuthology.orchestra.run.smithi119.stdout:uid [ unknown] Ceph.com (release key) 2022-04-09T11:40:33.779 DEBUG:teuthology.orchestra.run.smithi119:> sudo lsb_release -is 2022-04-09T11:40:33.827 INFO:teuthology.orchestra.run.smithi119.stdout:Ubuntu 2022-04-09T11:40:33.834 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-04-09T11:40:33.834 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2022-04-09T11:40:33.835 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=a70e525ea98f7194c5e516757c6105ca14c81cb7 2022-04-09T11:40:33.988 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default/ 2022-04-09T11:40:33.989 INFO:teuthology.task.install.deb:Package version is 17.0.0-11488-ga70e525e-1focal 2022-04-09T11:40:34.108 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-04-09T11:40:34.109 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2022-04-09T11:40:34.124 DEBUG:teuthology.orchestra.run.smithi119:> sudo apt-get update 2022-04-09T11:40:34.327 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-04-09T11:40:34.380 INFO:teuthology.orchestra.run.smithi119.stdout:Ign:2 https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default focal InRelease 2022-04-09T11:40:34.410 INFO:teuthology.orchestra.run.smithi119.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default focal Release [7,662 B] 2022-04-09T11:40:34.439 INFO:teuthology.orchestra.run.smithi119.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default focal Release.gpg 2022-04-09T11:40:34.454 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:6 http://archive.ubuntu.com/ubuntu focal InRelease 2022-04-09T11:40:34.460 INFO:teuthology.orchestra.run.smithi119.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default focal/main i386 Packages [3,355 B] 2022-04-09T11:40:34.496 INFO:teuthology.orchestra.run.smithi119.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2022-04-09T11:40:34.534 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-04-09T11:40:34.614 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:10 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-04-09T11:40:35.326 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:3 https://packages.cloud.google.com/apt kubernetes-xenial InRelease 2022-04-09T11:40:35.522 INFO:teuthology.orchestra.run.smithi119.stdout:Fetched 28.4 kB in 1s (24.5 kB/s) 2022-04-09T11:40:36.478 INFO:teuthology.orchestra.run.smithi119.stdout:Reading package lists... 2022-04-09T11:40:36.497 DEBUG:teuthology.orchestra.run.smithi119:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.0.0-11488-ga70e525e-1focal cephadm=17.0.0-11488-ga70e525e-1focal ceph-mds=17.0.0-11488-ga70e525e-1focal ceph-mgr=17.0.0-11488-ga70e525e-1focal ceph-common=17.0.0-11488-ga70e525e-1focal ceph-fuse=17.0.0-11488-ga70e525e-1focal ceph-test=17.0.0-11488-ga70e525e-1focal ceph-volume=17.0.0-11488-ga70e525e-1focal radosgw=17.0.0-11488-ga70e525e-1focal python3-rados=17.0.0-11488-ga70e525e-1focal python3-rgw=17.0.0-11488-ga70e525e-1focal python3-cephfs=17.0.0-11488-ga70e525e-1focal python3-rbd=17.0.0-11488-ga70e525e-1focal libcephfs2=17.0.0-11488-ga70e525e-1focal libcephfs-dev=17.0.0-11488-ga70e525e-1focal librados2=17.0.0-11488-ga70e525e-1focal librbd1=17.0.0-11488-ga70e525e-1focal rbd-fuse=17.0.0-11488-ga70e525e-1focal 2022-04-09T11:40:36.558 INFO:teuthology.orchestra.run.smithi119.stdout:Reading package lists... 2022-04-09T11:40:36.747 INFO:teuthology.orchestra.run.smithi119.stdout:Building dependency tree... 2022-04-09T11:40:36.747 INFO:teuthology.orchestra.run.smithi119.stdout:Reading state information... 2022-04-09T11:40:36.911 INFO:teuthology.orchestra.run.smithi119.stdout:The following packages were automatically installed and are no longer required: 2022-04-09T11:40:36.912 INFO:teuthology.orchestra.run.smithi119.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-04-09T11:40:36.912 INFO:teuthology.orchestra.run.smithi119.stdout:Use 'sudo apt autoremove' to remove them. 2022-04-09T11:40:36.912 INFO:teuthology.orchestra.run.smithi119.stdout:The following additional packages will be installed: 2022-04-09T11:40:36.913 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2022-04-09T11:40:36.913 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2022-04-09T11:40:36.913 INFO:teuthology.orchestra.run.smithi119.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2022-04-09T11:40:36.913 INFO:teuthology.orchestra.run.smithi119.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2022-04-09T11:40:36.913 INFO:teuthology.orchestra.run.smithi119.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2022-04-09T11:40:36.914 INFO:teuthology.orchestra.run.smithi119.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 libxslt1.1 nvme-cli 2022-04-09T11:40:36.915 INFO:teuthology.orchestra.run.smithi119.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2022-04-09T11:40:36.915 INFO:teuthology.orchestra.run.smithi119.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2022-04-09T11:40:36.915 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2022-04-09T11:40:36.915 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2022-04-09T11:40:36.916 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2022-04-09T11:40:36.916 INFO:teuthology.orchestra.run.smithi119.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2022-04-09T11:40:36.916 INFO:teuthology.orchestra.run.smithi119.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2022-04-09T11:40:36.916 INFO:teuthology.orchestra.run.smithi119.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2022-04-09T11:40:36.916 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2022-04-09T11:40:36.917 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2022-04-09T11:40:36.917 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2022-04-09T11:40:36.917 INFO:teuthology.orchestra.run.smithi119.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2022-04-09T11:40:36.917 INFO:teuthology.orchestra.run.smithi119.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2022-04-09T11:40:36.917 INFO:teuthology.orchestra.run.smithi119.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2022-04-09T11:40:36.918 INFO:teuthology.orchestra.run.smithi119.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2022-04-09T11:40:36.918 INFO:teuthology.orchestra.run.smithi119.stdout: xmlstarlet 2022-04-09T11:40:36.918 INFO:teuthology.orchestra.run.smithi119.stdout:Suggested packages: 2022-04-09T11:40:36.918 INFO:teuthology.orchestra.run.smithi119.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2022-04-09T11:40:36.919 INFO:teuthology.orchestra.run.smithi119.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2022-04-09T11:40:36.919 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2022-04-09T11:40:36.919 INFO:teuthology.orchestra.run.smithi119.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2022-04-09T11:40:36.919 INFO:teuthology.orchestra.run.smithi119.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2022-04-09T11:40:36.919 INFO:teuthology.orchestra.run.smithi119.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2022-04-09T11:40:36.920 INFO:teuthology.orchestra.run.smithi119.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2022-04-09T11:40:36.920 INFO:teuthology.orchestra.run.smithi119.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2022-04-09T11:40:36.920 INFO:teuthology.orchestra.run.smithi119.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2022-04-09T11:40:36.920 INFO:teuthology.orchestra.run.smithi119.stdout:Recommended packages: 2022-04-09T11:40:36.920 INFO:teuthology.orchestra.run.smithi119.stdout: btrfs-tools 2022-04-09T11:40:37.036 INFO:teuthology.orchestra.run.smithi119.stdout:The following NEW packages will be installed: 2022-04-09T11:40:37.037 INFO:teuthology.orchestra.run.smithi119.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2022-04-09T11:40:37.037 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2022-04-09T11:40:37.037 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2022-04-09T11:40:37.037 INFO:teuthology.orchestra.run.smithi119.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2022-04-09T11:40:37.038 INFO:teuthology.orchestra.run.smithi119.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2022-04-09T11:40:37.038 INFO:teuthology.orchestra.run.smithi119.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2022-04-09T11:40:37.038 INFO:teuthology.orchestra.run.smithi119.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2022-04-09T11:40:37.040 INFO:teuthology.orchestra.run.smithi119.stdout: libsqlite3-mod-ceph libthrift-0.13.0 libxslt1.1 nvme-cli python-natsort-doc 2022-04-09T11:40:37.040 INFO:teuthology.orchestra.run.smithi119.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2022-04-09T11:40:37.040 INFO:teuthology.orchestra.run.smithi119.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2022-04-09T11:40:37.040 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2022-04-09T11:40:37.041 INFO:teuthology.orchestra.run.smithi119.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2022-04-09T11:40:37.041 INFO:teuthology.orchestra.run.smithi119.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2022-04-09T11:40:37.041 INFO:teuthology.orchestra.run.smithi119.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2022-04-09T11:40:37.041 INFO:teuthology.orchestra.run.smithi119.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2022-04-09T11:40:37.041 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2022-04-09T11:40:37.042 INFO:teuthology.orchestra.run.smithi119.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2022-04-09T11:40:37.042 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2022-04-09T11:40:37.042 INFO:teuthology.orchestra.run.smithi119.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2022-04-09T11:40:37.042 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2022-04-09T11:40:37.043 INFO:teuthology.orchestra.run.smithi119.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2022-04-09T11:40:37.043 INFO:teuthology.orchestra.run.smithi119.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2022-04-09T11:40:37.043 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2022-04-09T11:40:37.043 INFO:teuthology.orchestra.run.smithi119.stdout: qttranslations5-l10n radosgw rbd-fuse xmlstarlet 2022-04-09T11:40:37.043 INFO:teuthology.orchestra.run.smithi119.stdout:The following packages will be upgraded: 2022-04-09T11:40:37.044 INFO:teuthology.orchestra.run.smithi119.stdout: librados2 librbd1 2022-04-09T11:40:37.225 INFO:teuthology.orchestra.run.smithi119.stdout:2 upgraded, 108 newly installed, 0 to remove and 8 not upgraded. 2022-04-09T11:40:37.225 INFO:teuthology.orchestra.run.smithi119.stdout:Need to get 209 MB of archives. 2022-04-09T11:40:37.226 INFO:teuthology.orchestra.run.smithi119.stdout:After this operation, 979 MB of additional disk space will be used. 2022-04-09T11:40:37.226 INFO:teuthology.orchestra.run.smithi119.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2022-04-09T11:40:37.307 INFO:teuthology.orchestra.run.smithi119.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.0.0-11488-ga70e525e-1focal [3,482 kB] 2022-04-09T11:40:37.621 INFO:teuthology.orchestra.run.smithi119.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2022-04-09T11:40:37.633 INFO:teuthology.orchestra.run.smithi119.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libpcre2-16-0 amd64 10.34-7 [181 kB] 2022-04-09T11:40:37.717 INFO:teuthology.orchestra.run.smithi119.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2022-04-09T11:40:37.955 INFO:teuthology.orchestra.run.smithi119.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2022-04-09T11:40:37.965 INFO:teuthology.orchestra.run.smithi119.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2022-04-09T11:40:37.993 INFO:teuthology.orchestra.run.smithi119.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2022-04-09T11:40:38.001 INFO:teuthology.orchestra.run.smithi119.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2022-04-09T11:40:38.002 INFO:teuthology.orchestra.run.smithi119.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2022-04-09T11:40:38.006 INFO:teuthology.orchestra.run.smithi119.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2022-04-09T11:40:38.007 INFO:teuthology.orchestra.run.smithi119.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2022-04-09T11:40:38.019 INFO:teuthology.orchestra.run.smithi119.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2022-04-09T11:40:38.020 INFO:teuthology.orchestra.run.smithi119.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2022-04-09T11:40:38.036 INFO:teuthology.orchestra.run.smithi119.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2022-04-09T11:40:38.058 INFO:teuthology.orchestra.run.smithi119.stdout:Get:16 https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.0.0-11488-ga70e525e-1focal [3,708 kB] 2022-04-09T11:40:38.107 INFO:teuthology.orchestra.run.smithi119.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2022-04-09T11:40:38.108 INFO:teuthology.orchestra.run.smithi119.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2022-04-09T11:40:38.111 INFO:teuthology.orchestra.run.smithi119.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2022-04-09T11:40:38.112 INFO:teuthology.orchestra.run.smithi119.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jwt all 1.7.1-2ubuntu2 [17.4 kB] 2022-04-09T11:40:38.113 INFO:teuthology.orchestra.run.smithi119.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2022-04-09T11:40:38.114 INFO:teuthology.orchestra.run.smithi119.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2022-04-09T11:40:38.114 INFO:teuthology.orchestra.run.smithi119.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2022-04-09T11:40:38.116 INFO:teuthology.orchestra.run.smithi119.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2022-04-09T11:40:38.117 INFO:teuthology.orchestra.run.smithi119.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2022-04-09T11:40:38.196 INFO:teuthology.orchestra.run.smithi119.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2022-04-09T11:40:38.197 INFO:teuthology.orchestra.run.smithi119.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2022-04-09T11:40:38.199 INFO:teuthology.orchestra.run.smithi119.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2022-04-09T11:40:38.201 INFO:teuthology.orchestra.run.smithi119.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2022-04-09T11:40:38.204 INFO:teuthology.orchestra.run.smithi119.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2022-04-09T11:40:38.204 INFO:teuthology.orchestra.run.smithi119.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2022-04-09T11:40:38.276 INFO:teuthology.orchestra.run.smithi119.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2022-04-09T11:40:38.276 INFO:teuthology.orchestra.run.smithi119.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2022-04-09T11:40:38.277 INFO:teuthology.orchestra.run.smithi119.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2022-04-09T11:40:38.277 INFO:teuthology.orchestra.run.smithi119.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2022-04-09T11:40:38.355 INFO:teuthology.orchestra.run.smithi119.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2022-04-09T11:40:38.358 INFO:teuthology.orchestra.run.smithi119.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2022-04-09T11:40:38.360 INFO:teuthology.orchestra.run.smithi119.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2022-04-09T11:40:38.361 INFO:teuthology.orchestra.run.smithi119.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2022-04-09T11:40:38.362 INFO:teuthology.orchestra.run.smithi119.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2022-04-09T11:40:38.364 INFO:teuthology.orchestra.run.smithi119.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2022-04-09T11:40:38.435 INFO:teuthology.orchestra.run.smithi119.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2022-04-09T11:40:38.436 INFO:teuthology.orchestra.run.smithi119.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2022-04-09T11:40:38.436 INFO:teuthology.orchestra.run.smithi119.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2022-04-09T11:40:38.509 INFO:teuthology.orchestra.run.smithi119.stdout:Get:45 https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.0.0-11488-ga70e525e-1focal [755 kB] 2022-04-09T11:40:38.592 INFO:teuthology.orchestra.run.smithi119.stdout:Get:46 https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.0.0-11488-ga70e525e-1focal [413 kB] 2022-04-09T11:40:38.634 INFO:teuthology.orchestra.run.smithi119.stdout:Get:47 https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.0.0-11488-ga70e525e-1focal [22.4 kB] 2022-04-09T11:40:38.637 INFO:teuthology.orchestra.run.smithi119.stdout:Get:48 https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.0.0-11488-ga70e525e-1focal [200 kB] 2022-04-09T11:40:38.667 INFO:teuthology.orchestra.run.smithi119.stdout:Get:49 https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.0.0-11488-ga70e525e-1focal [42.4 kB] 2022-04-09T11:40:38.676 INFO:teuthology.orchestra.run.smithi119.stdout:Get:50 https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.0.0-11488-ga70e525e-1focal [385 kB] 2022-04-09T11:40:38.707 INFO:teuthology.orchestra.run.smithi119.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2022-04-09T11:40:38.756 INFO:teuthology.orchestra.run.smithi119.stdout:Get:52 https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.0.0-11488-ga70e525e-1focal [5,536 kB] 2022-04-09T11:40:38.784 INFO:teuthology.orchestra.run.smithi119.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2022-04-09T11:40:38.786 INFO:teuthology.orchestra.run.smithi119.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2022-04-09T11:40:38.842 INFO:teuthology.orchestra.run.smithi119.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2022-04-09T11:40:38.843 INFO:teuthology.orchestra.run.smithi119.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2022-04-09T11:40:38.843 INFO:teuthology.orchestra.run.smithi119.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2022-04-09T11:40:38.844 INFO:teuthology.orchestra.run.smithi119.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2022-04-09T11:40:38.844 INFO:teuthology.orchestra.run.smithi119.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2022-04-09T11:40:38.852 INFO:teuthology.orchestra.run.smithi119.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2022-04-09T11:40:38.853 INFO:teuthology.orchestra.run.smithi119.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2022-04-09T11:40:38.868 INFO:teuthology.orchestra.run.smithi119.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2022-04-09T11:40:38.868 INFO:teuthology.orchestra.run.smithi119.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2022-04-09T11:40:38.948 INFO:teuthology.orchestra.run.smithi119.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2022-04-09T11:40:38.954 INFO:teuthology.orchestra.run.smithi119.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2022-04-09T11:40:38.957 INFO:teuthology.orchestra.run.smithi119.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2022-04-09T11:40:38.959 INFO:teuthology.orchestra.run.smithi119.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2022-04-09T11:40:38.960 INFO:teuthology.orchestra.run.smithi119.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2022-04-09T11:40:38.961 INFO:teuthology.orchestra.run.smithi119.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2022-04-09T11:40:38.963 INFO:teuthology.orchestra.run.smithi119.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2022-04-09T11:40:39.030 INFO:teuthology.orchestra.run.smithi119.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2022-04-09T11:40:39.035 INFO:teuthology.orchestra.run.smithi119.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2022-04-09T11:40:39.197 INFO:teuthology.orchestra.run.smithi119.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2022-04-09T11:40:39.199 INFO:teuthology.orchestra.run.smithi119.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2022-04-09T11:40:39.200 INFO:teuthology.orchestra.run.smithi119.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2022-04-09T11:40:39.200 INFO:teuthology.orchestra.run.smithi119.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2022-04-09T11:40:39.201 INFO:teuthology.orchestra.run.smithi119.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2022-04-09T11:40:39.202 INFO:teuthology.orchestra.run.smithi119.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2022-04-09T11:40:39.219 INFO:teuthology.orchestra.run.smithi119.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2022-04-09T11:40:39.219 INFO:teuthology.orchestra.run.smithi119.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2022-04-09T11:40:39.220 INFO:teuthology.orchestra.run.smithi119.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2022-04-09T11:40:39.398 INFO:teuthology.orchestra.run.smithi119.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2022-04-09T11:40:39.400 INFO:teuthology.orchestra.run.smithi119.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2022-04-09T11:40:39.401 INFO:teuthology.orchestra.run.smithi119.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2022-04-09T11:40:39.402 INFO:teuthology.orchestra.run.smithi119.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2022-04-09T11:40:39.402 INFO:teuthology.orchestra.run.smithi119.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2022-04-09T11:40:39.403 INFO:teuthology.orchestra.run.smithi119.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2022-04-09T11:40:39.404 INFO:teuthology.orchestra.run.smithi119.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2022-04-09T11:40:39.580 INFO:teuthology.orchestra.run.smithi119.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.0.0-11488-ga70e525e-1focal [116 kB] 2022-04-09T11:40:39.583 INFO:teuthology.orchestra.run.smithi119.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.0.0-11488-ga70e525e-1focal [477 kB] 2022-04-09T11:40:39.654 INFO:teuthology.orchestra.run.smithi119.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.0.0-11488-ga70e525e-1focal [27.3 MB] 2022-04-09T11:40:42.388 INFO:teuthology.orchestra.run.smithi119.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.0.0-11488-ga70e525e-1focal [6,430 kB] 2022-04-09T11:40:43.374 INFO:teuthology.orchestra.run.smithi119.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.0.0-11488-ga70e525e-1focal [212 kB] 2022-04-09T11:40:43.434 INFO:teuthology.orchestra.run.smithi119.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.0.0-11488-ga70e525e-1focal [152 kB] 2022-04-09T11:40:43.520 INFO:teuthology.orchestra.run.smithi119.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.0.0-11488-ga70e525e-1focal [1,574 kB] 2022-04-09T11:40:43.821 INFO:teuthology.orchestra.run.smithi119.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.0.0-11488-ga70e525e-1focal [7,803 kB] 2022-04-09T11:40:45.259 INFO:teuthology.orchestra.run.smithi119.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.0.0-11488-ga70e525e-1focal [28.8 MB] 2022-04-09T11:40:52.143 INFO:teuthology.orchestra.run.smithi119.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.0.0-11488-ga70e525e-1focal [3,812 B] 2022-04-09T11:40:52.143 INFO:teuthology.orchestra.run.smithi119.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.0.0-11488-ga70e525e-1focal [865 kB] 2022-04-09T11:40:52.211 INFO:teuthology.orchestra.run.smithi119.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.0.0-11488-ga70e525e-1focal [2,396 kB] 2022-04-09T11:40:52.477 INFO:teuthology.orchestra.run.smithi119.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.0.0-11488-ga70e525e-1focal [70.1 kB] 2022-04-09T11:40:52.495 INFO:teuthology.orchestra.run.smithi119.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.0.0-11488-ga70e525e-1focal [100 kB] 2022-04-09T11:40:52.498 INFO:teuthology.orchestra.run.smithi119.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.0.0-11488-ga70e525e-1focal [1,678 kB] 2022-04-09T11:40:52.679 INFO:teuthology.orchestra.run.smithi119.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.0.0-11488-ga70e525e-1focal [7,900 kB] 2022-04-09T11:40:53.539 INFO:teuthology.orchestra.run.smithi119.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.0.0-11488-ga70e525e-1focal [3,928 B] 2022-04-09T11:40:53.539 INFO:teuthology.orchestra.run.smithi119.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.0.0-11488-ga70e525e-1focal [56.3 MB] 2022-04-09T11:41:00.100 INFO:teuthology.orchestra.run.smithi119.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.0.0-11488-ga70e525e-1focal [220 kB] 2022-04-09T11:41:00.104 INFO:teuthology.orchestra.run.smithi119.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.0.0-11488-ga70e525e-1focal [21.7 kB] 2022-04-09T11:41:00.105 INFO:teuthology.orchestra.run.smithi119.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.0.0-11488-ga70e525e-1focal [16.9 MB] 2022-04-09T11:41:01.844 INFO:teuthology.orchestra.run.smithi119.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-omri-tracing-compiled/a70e525ea98f7194c5e516757c6105ca14c81cb7/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.0.0-11488-ga70e525e-1focal [78.8 kB] 2022-04-09T11:41:02.474 INFO:teuthology.orchestra.run.smithi119.stdout:Fetched 209 MB in 25s (8,418 kB/s) 2022-04-09T11:41:02.666 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2022-04-09T11:41:02.713 INFO:teuthology.orchestra.run.smithi119.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109467 files and directories currently installed.) 2022-04-09T11:41:02.716 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2022-04-09T11:41:02.874 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2022-04-09T11:41:03.303 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2022-04-09T11:41:03.315 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2022-04-09T11:41:03.353 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2022-04-09T11:41:03.740 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2022-04-09T11:41:03.751 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../002-libpcre2-16-0_10.34-7_amd64.deb ... 2022-04-09T11:41:03.801 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7) ... 2022-04-09T11:41:04.432 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libqt5core5a:amd64. 2022-04-09T11:41:04.445 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../003-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2022-04-09T11:41:04.504 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2022-04-09T11:41:05.041 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2022-04-09T11:41:05.057 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../004-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2022-04-09T11:41:05.098 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2022-04-09T11:41:05.545 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libqt5network5:amd64. 2022-04-09T11:41:05.557 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../005-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2022-04-09T11:41:05.594 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2022-04-09T11:41:06.123 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libthrift-0.13.0. 2022-04-09T11:41:06.131 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../006-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2022-04-09T11:41:06.193 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2022-04-09T11:41:06.650 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../007-librbd1_17.0.0-11488-ga70e525e-1focal_amd64.deb ... 2022-04-09T11:41:06.778 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking librbd1 (17.0.0-11488-ga70e525e-1focal) over (15.2.14-0ubuntu0.20.04.2) ... 2022-04-09T11:41:08.845 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../008-librados2_17.0.0-11488-ga70e525e-1focal_amd64.deb ... 2022-04-09T11:41:09.011 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking librados2 (17.0.0-11488-ga70e525e-1focal) over (15.2.14-0ubuntu0.20.04.2) ... 2022-04-09T11:41:09.749 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libcephfs2. 2022-04-09T11:41:09.757 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../009-libcephfs2_17.0.0-11488-ga70e525e-1focal_amd64.deb ... 2022-04-09T11:41:09.799 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libcephfs2 (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:41:10.293 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-rados. 2022-04-09T11:41:10.305 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../010-python3-rados_17.0.0-11488-ga70e525e-1focal_amd64.deb ... 2022-04-09T11:41:10.341 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-rados (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:41:10.822 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-ceph-argparse. 2022-04-09T11:41:10.837 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../011-python3-ceph-argparse_17.0.0-11488-ga70e525e-1focal_all.deb ... 2022-04-09T11:41:10.888 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-ceph-argparse (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:41:11.201 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-cephfs. 2022-04-09T11:41:11.213 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../012-python3-cephfs_17.0.0-11488-ga70e525e-1focal_amd64.deb ... 2022-04-09T11:41:11.297 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-cephfs (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:41:11.714 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-ceph-common. 2022-04-09T11:41:11.724 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../013-python3-ceph-common_17.0.0-11488-ga70e525e-1focal_all.deb ... 2022-04-09T11:41:11.764 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-ceph-common (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:41:12.126 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-prettytable. 2022-04-09T11:41:12.138 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../014-python3-prettytable_0.7.2-5_all.deb ... 2022-04-09T11:41:12.167 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2022-04-09T11:41:12.437 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-rbd. 2022-04-09T11:41:12.449 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../015-python3-rbd_17.0.0-11488-ga70e525e-1focal_amd64.deb ... 2022-04-09T11:41:12.524 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-rbd (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:41:12.975 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2022-04-09T11:41:12.986 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../016-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2022-04-09T11:41:13.075 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2022-04-09T11:41:13.446 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package librabbitmq4:amd64. 2022-04-09T11:41:13.459 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../017-librabbitmq4_0.10.0-1_amd64.deb ... 2022-04-09T11:41:13.495 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2022-04-09T11:41:13.942 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package librdkafka1:amd64. 2022-04-09T11:41:13.958 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../018-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2022-04-09T11:41:14.033 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-04-09T11:41:14.429 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package librgw2. 2022-04-09T11:41:14.442 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../019-librgw2_17.0.0-11488-ga70e525e-1focal_amd64.deb ... 2022-04-09T11:41:14.512 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking librgw2 (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:41:15.333 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-rgw. 2022-04-09T11:41:15.344 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../020-python3-rgw_17.0.0-11488-ga70e525e-1focal_amd64.deb ... 2022-04-09T11:41:15.374 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-rgw (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:41:16.005 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package liboath0. 2022-04-09T11:41:16.016 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2022-04-09T11:41:16.096 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2022-04-09T11:41:16.466 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libradosstriper1. 2022-04-09T11:41:16.480 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../022-libradosstriper1_17.0.0-11488-ga70e525e-1focal_amd64.deb ... 2022-04-09T11:41:16.558 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libradosstriper1 (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:41:17.020 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-common. 2022-04-09T11:41:17.032 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../023-ceph-common_17.0.0-11488-ga70e525e-1focal_amd64.deb ... 2022-04-09T11:41:17.094 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-common (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:41:19.972 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-base. 2022-04-09T11:41:19.986 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../024-ceph-base_17.0.0-11488-ga70e525e-1focal_amd64.deb ... 2022-04-09T11:41:20.037 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-base (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:41:21.229 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-cryptography. 2022-04-09T11:41:21.242 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2022-04-09T11:41:21.287 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2022-04-09T11:41:21.691 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-openssl. 2022-04-09T11:41:21.701 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2022-04-09T11:41:21.741 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2022-04-09T11:41:22.212 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2022-04-09T11:41:22.225 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../027-ceph-mgr-modules-core_17.0.0-11488-ga70e525e-1focal_all.deb ... 2022-04-09T11:41:22.270 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-mgr-modules-core (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:41:22.700 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-bcrypt. 2022-04-09T11:41:22.712 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../028-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2022-04-09T11:41:22.750 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2022-04-09T11:41:23.163 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-cherrypy3. 2022-04-09T11:41:23.170 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../029-python3-cherrypy3_8.9.1-6_all.deb ... 2022-04-09T11:41:23.230 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2022-04-09T11:41:23.575 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-natsort. 2022-04-09T11:41:23.587 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../030-python3-natsort_7.0.1-1_all.deb ... 2022-04-09T11:41:23.658 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-natsort (7.0.1-1) ... 2022-04-09T11:41:23.997 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-jwt. 2022-04-09T11:41:24.009 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../031-python3-jwt_1.7.1-2ubuntu2_all.deb ... 2022-04-09T11:41:24.089 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2) ... 2022-04-09T11:41:24.599 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-logutils. 2022-04-09T11:41:24.610 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../032-python3-logutils_0.3.3-7_all.deb ... 2022-04-09T11:41:24.652 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-logutils (0.3.3-7) ... 2022-04-09T11:41:24.997 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-markupsafe. 2022-04-09T11:41:25.009 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../033-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2022-04-09T11:41:25.047 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2022-04-09T11:41:25.367 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-mako. 2022-04-09T11:41:25.380 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../034-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2022-04-09T11:41:25.423 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-04-09T11:41:25.838 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-simplegeneric. 2022-04-09T11:41:25.846 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../035-python3-simplegeneric_0.8.1-3_all.deb ... 2022-04-09T11:41:25.913 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2022-04-09T11:41:26.266 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-singledispatch. 2022-04-09T11:41:26.278 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../036-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2022-04-09T11:41:26.316 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2022-04-09T11:41:26.736 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-webob. 2022-04-09T11:41:26.749 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../037-python3-webob_1%3a1.8.5-2_all.deb ... 2022-04-09T11:41:26.778 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2022-04-09T11:41:27.123 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-soupsieve. 2022-04-09T11:41:27.138 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../038-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2022-04-09T11:41:27.181 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2022-04-09T11:41:27.517 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-bs4. 2022-04-09T11:41:27.530 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../039-python3-bs4_4.8.2-1_all.deb ... 2022-04-09T11:41:27.567 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2022-04-09T11:41:28.182 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-waitress. 2022-04-09T11:41:28.191 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../040-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2022-04-09T11:41:28.233 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2022-04-09T11:41:28.609 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-tempita. 2022-04-09T11:41:28.623 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../041-python3-tempita_0.5.2-6_all.deb ... 2022-04-09T11:41:28.667 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-tempita (0.5.2-6) ... 2022-04-09T11:41:29.006 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-paste. 2022-04-09T11:41:29.019 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../042-python3-paste_3.4.0+dfsg1-1_all.deb ... 2022-04-09T11:41:29.096 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2022-04-09T11:41:29.534 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2022-04-09T11:41:29.546 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../043-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2022-04-09T11:41:29.583 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2022-04-09T11:41:29.912 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-pastedeploy. 2022-04-09T11:41:29.925 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../044-python3-pastedeploy_2.1.0-1_all.deb ... 2022-04-09T11:41:29.969 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2022-04-09T11:41:30.357 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-webtest. 2022-04-09T11:41:30.373 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../045-python3-webtest_2.0.32-2_all.deb ... 2022-04-09T11:41:30.449 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-webtest (2.0.32-2) ... 2022-04-09T11:41:30.870 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-pecan. 2022-04-09T11:41:30.886 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../046-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2022-04-09T11:41:30.928 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2022-04-09T11:41:31.376 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-werkzeug. 2022-04-09T11:41:31.384 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../047-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2022-04-09T11:41:31.433 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2022-04-09T11:41:31.880 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2022-04-09T11:41:31.892 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../048-libsqlite3-mod-ceph_17.0.0-11488-ga70e525e-1focal_amd64.deb ... 2022-04-09T11:41:31.929 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libsqlite3-mod-ceph (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:41:32.291 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-mgr. 2022-04-09T11:41:32.304 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../049-ceph-mgr_17.0.0-11488-ga70e525e-1focal_amd64.deb ... 2022-04-09T11:41:32.341 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-mgr (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:41:32.970 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-mon. 2022-04-09T11:41:32.984 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../050-ceph-mon_17.0.0-11488-ga70e525e-1focal_amd64.deb ... 2022-04-09T11:41:33.028 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-mon (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:41:34.461 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-osd. 2022-04-09T11:41:34.473 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../051-ceph-osd_17.0.0-11488-ga70e525e-1focal_amd64.deb ... 2022-04-09T11:41:34.576 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-osd (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:41:37.583 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph. 2022-04-09T11:41:37.595 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../052-ceph_17.0.0-11488-ga70e525e-1focal_amd64.deb ... 2022-04-09T11:41:37.632 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:41:38.179 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-fuse. 2022-04-09T11:41:38.191 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../053-ceph-fuse_17.0.0-11488-ga70e525e-1focal_amd64.deb ... 2022-04-09T11:41:38.263 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-fuse (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:41:38.741 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-mds. 2022-04-09T11:41:38.766 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../054-ceph-mds_17.0.0-11488-ga70e525e-1focal_amd64.deb ... 2022-04-09T11:41:38.799 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-mds (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:41:39.420 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package cephadm. 2022-04-09T11:41:39.432 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../055-cephadm_17.0.0-11488-ga70e525e-1focal_amd64.deb ... 2022-04-09T11:41:39.469 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking cephadm (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:41:40.750 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-jinja2. 2022-04-09T11:41:40.766 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../056-python3-jinja2_2.10.1-2_all.deb ... 2022-04-09T11:41:40.816 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2022-04-09T11:41:41.195 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2022-04-09T11:41:41.207 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../057-ceph-mgr-cephadm_17.0.0-11488-ga70e525e-1focal_all.deb ... 2022-04-09T11:41:41.261 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-mgr-cephadm (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:41:41.730 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-repoze.lru. 2022-04-09T11:41:41.742 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../058-python3-repoze.lru_0.7-2_all.deb ... 2022-04-09T11:41:41.808 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2022-04-09T11:41:42.078 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libjs-underscore. 2022-04-09T11:41:42.091 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../059-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2022-04-09T11:41:42.137 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-04-09T11:41:42.466 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libjs-sphinxdoc. 2022-04-09T11:41:42.475 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../060-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2022-04-09T11:41:42.524 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-04-09T11:41:42.953 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-routes. 2022-04-09T11:41:42.966 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../061-python3-routes_2.4.1-2_all.deb ... 2022-04-09T11:41:43.002 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-routes (2.4.1-2) ... 2022-04-09T11:41:43.365 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2022-04-09T11:41:43.379 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../062-ceph-mgr-dashboard_17.0.0-11488-ga70e525e-1focal_all.deb ... 2022-04-09T11:41:43.423 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-mgr-dashboard (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:41:44.069 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-decorator. 2022-04-09T11:41:44.079 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../063-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2022-04-09T11:41:44.142 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2022-04-09T11:41:44.716 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2022-04-09T11:41:44.725 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../064-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2022-04-09T11:41:44.767 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-04-09T11:41:45.099 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-scipy. 2022-04-09T11:41:45.109 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../065-python3-scipy_1.3.3-3build1_amd64.deb ... 2022-04-09T11:41:45.153 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2022-04-09T11:41:47.019 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-sklearn-lib. 2022-04-09T11:41:47.031 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../066-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2022-04-09T11:41:47.077 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-04-09T11:41:47.757 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-joblib. 2022-04-09T11:41:47.771 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../067-python3-joblib_0.14.0-3_all.deb ... 2022-04-09T11:41:47.823 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-joblib (0.14.0-3) ... 2022-04-09T11:41:48.285 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-sklearn. 2022-04-09T11:41:48.299 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../068-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2022-04-09T11:41:48.343 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-04-09T11:41:49.254 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2022-04-09T11:41:49.269 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../069-ceph-mgr-diskprediction-local_17.0.0-11488-ga70e525e-1focal_all.deb ... 2022-04-09T11:41:49.321 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-mgr-diskprediction-local (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:41:50.534 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-cachetools. 2022-04-09T11:41:50.545 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../070-python3-cachetools_4.0.0-1_all.deb ... 2022-04-09T11:41:50.602 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2022-04-09T11:41:51.013 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-pyasn1. 2022-04-09T11:41:51.027 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../071-python3-pyasn1_0.4.2-3build1_all.deb ... 2022-04-09T11:41:51.071 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2022-04-09T11:41:51.535 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-pyasn1-modules. 2022-04-09T11:41:51.548 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../072-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2022-04-09T11:41:51.584 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-04-09T11:41:51.938 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-rsa. 2022-04-09T11:41:51.955 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../073-python3-rsa_4.0-3_all.deb ... 2022-04-09T11:41:51.996 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-rsa (4.0-3) ... 2022-04-09T11:41:52.325 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-google-auth. 2022-04-09T11:41:52.333 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../074-python3-google-auth_1.5.1-2_all.deb ... 2022-04-09T11:41:52.391 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2022-04-09T11:41:52.830 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-blinker. 2022-04-09T11:41:52.844 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../075-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2022-04-09T11:41:52.922 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-04-09T11:41:53.208 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-oauthlib. 2022-04-09T11:41:53.216 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../076-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2022-04-09T11:41:53.283 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2022-04-09T11:41:53.704 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-requests-oauthlib. 2022-04-09T11:41:53.714 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../077-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2022-04-09T11:41:53.754 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-04-09T11:41:54.177 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-websocket. 2022-04-09T11:41:54.190 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../078-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2022-04-09T11:41:54.242 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2022-04-09T11:41:54.712 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-kubernetes. 2022-04-09T11:41:54.727 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../079-python3-kubernetes_7.0.0~a1-2_all.deb ... 2022-04-09T11:41:54.821 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2022-04-09T11:41:55.446 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2022-04-09T11:41:55.456 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../080-ceph-mgr-k8sevents_17.0.0-11488-ga70e525e-1focal_all.deb ... 2022-04-09T11:41:55.495 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-mgr-k8sevents (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:41:56.148 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libonig5:amd64. 2022-04-09T11:41:56.161 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../081-libonig5_6.9.4-1_amd64.deb ... 2022-04-09T11:41:56.223 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2022-04-09T11:41:56.639 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libjq1:amd64. 2022-04-09T11:41:56.653 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../082-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-04-09T11:41:56.705 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-04-09T11:41:57.054 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package jq. 2022-04-09T11:41:57.067 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../083-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-04-09T11:41:57.105 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2022-04-09T11:41:57.497 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libxslt1.1:amd64. 2022-04-09T11:41:57.507 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../084-libxslt1.1_1.1.34-4_amd64.deb ... 2022-04-09T11:41:57.546 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2022-04-09T11:41:57.878 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package xmlstarlet. 2022-04-09T11:41:57.891 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2build1_amd64.deb ... 2022-04-09T11:41:57.929 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2022-04-09T11:41:58.306 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-test. 2022-04-09T11:41:58.319 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../086-ceph-test_17.0.0-11488-ga70e525e-1focal_amd64.deb ... 2022-04-09T11:41:58.393 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-test (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:42:03.857 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-volume. 2022-04-09T11:42:03.868 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../087-ceph-volume_17.0.0-11488-ga70e525e-1focal_all.deb ... 2022-04-09T11:42:03.909 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-volume (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:42:04.336 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package fonts-mathjax. 2022-04-09T11:42:04.345 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../088-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2022-04-09T11:42:04.386 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2022-04-09T11:42:05.119 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libcephfs-dev. 2022-04-09T11:42:05.134 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../089-libcephfs-dev_17.0.0-11488-ga70e525e-1focal_amd64.deb ... 2022-04-09T11:42:05.177 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libcephfs-dev (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:42:05.780 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package nvme-cli. 2022-04-09T11:42:05.794 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../090-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2022-04-09T11:42:05.931 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2022-04-09T11:42:06.372 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libjs-mathjax. 2022-04-09T11:42:06.385 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../091-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2022-04-09T11:42:06.422 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2022-04-09T11:42:08.964 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python-natsort-doc. 2022-04-09T11:42:08.979 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../092-python-natsort-doc_7.0.1-1_all.deb ... 2022-04-09T11:42:09.041 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2022-04-09T11:42:09.561 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-atomicwrites. 2022-04-09T11:42:09.576 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../093-python3-atomicwrites_1.1.5-2build1_all.deb ... 2022-04-09T11:42:09.641 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2022-04-09T11:42:10.035 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-attr. 2022-04-09T11:42:10.049 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../094-python3-attr_19.3.0-2_all.deb ... 2022-04-09T11:42:10.102 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-attr (19.3.0-2) ... 2022-04-09T11:42:10.488 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-webencodings. 2022-04-09T11:42:10.500 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../095-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2022-04-09T11:42:10.549 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2022-04-09T11:42:11.099 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-html5lib. 2022-04-09T11:42:11.116 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../096-python3-html5lib_1.0.1-2_all.deb ... 2022-04-09T11:42:11.676 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2022-04-09T11:42:12.617 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-lxml:amd64. 2022-04-09T11:42:12.630 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../097-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2022-04-09T11:42:12.674 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-04-09T11:42:13.261 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-packaging. 2022-04-09T11:42:13.274 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../098-python3-packaging_20.3-1_all.deb ... 2022-04-09T11:42:13.380 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-packaging (20.3-1) ... 2022-04-09T11:42:13.821 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-pastescript. 2022-04-09T11:42:13.832 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2022-04-09T11:42:13.896 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2022-04-09T11:42:14.291 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-pluggy. 2022-04-09T11:42:14.304 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2022-04-09T11:42:14.366 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2022-04-09T11:42:14.844 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-psutil. 2022-04-09T11:42:14.858 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2022-04-09T11:42:14.893 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2022-04-09T11:42:15.253 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-py. 2022-04-09T11:42:15.263 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2022-04-09T11:42:15.311 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2022-04-09T11:42:15.727 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-pyinotify. 2022-04-09T11:42:15.741 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2022-04-09T11:42:15.793 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-04-09T11:42:16.203 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-wcwidth. 2022-04-09T11:42:16.212 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2022-04-09T11:42:16.253 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2022-04-09T11:42:16.671 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-pytest. 2022-04-09T11:42:16.688 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2022-04-09T11:42:16.752 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-pytest (4.6.9-1) ... 2022-04-09T11:42:17.129 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-simplejson. 2022-04-09T11:42:17.143 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2022-04-09T11:42:17.179 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2022-04-09T11:42:17.520 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package qttranslations5-l10n. 2022-04-09T11:42:17.535 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2022-04-09T11:42:17.577 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2022-04-09T11:42:18.643 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package radosgw. 2022-04-09T11:42:18.657 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../108-radosgw_17.0.0-11488-ga70e525e-1focal_amd64.deb ... 2022-04-09T11:42:18.709 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking radosgw (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:42:20.609 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package rbd-fuse. 2022-04-09T11:42:20.623 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../109-rbd-fuse_17.0.0-11488-ga70e525e-1focal_amd64.deb ... 2022-04-09T11:42:20.667 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking rbd-fuse (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:42:21.051 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-attr (19.3.0-2) ... 2022-04-09T11:42:21.426 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2022-04-09T11:42:21.566 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2022-04-09T11:42:21.693 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2022-04-09T11:42:21.886 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up cephadm (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:42:22.741 INFO:teuthology.orchestra.run.smithi119.stdout:Adding system user cephadm....done 2022-04-09T11:42:22.921 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2022-04-09T11:42:23.109 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2022-04-09T11:42:23.512 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-repoze.lru (0.7-2) ... 2022-04-09T11:42:23.899 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2022-04-09T11:42:24.060 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up liboath0 (2.6.1-1.3) ... 2022-04-09T11:42:24.271 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2022-04-09T11:42:24.694 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-joblib (0.14.0-3) ... 2022-04-09T11:42:25.142 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-jwt (1.7.1-2ubuntu2) ... 2022-04-09T11:42:25.447 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-cachetools (4.0.0-1) ... 2022-04-09T11:42:25.770 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2022-04-09T11:42:26.177 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-04-09T11:42:26.473 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-ceph-argparse (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:42:26.778 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-04-09T11:42:26.933 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2022-04-09T11:42:27.237 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2022-04-09T11:42:27.596 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libpcre2-16-0:amd64 (10.34-7) ... 2022-04-09T11:42:27.743 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2022-04-09T11:42:28.142 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-natsort (7.0.1-1) ... 2022-04-09T11:42:28.453 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2022-04-09T11:42:28.884 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2022-04-09T11:42:29.251 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2022-04-09T11:42:29.566 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2022-04-09T11:42:29.923 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-tempita (0.5.2-6) ... 2022-04-09T11:42:30.203 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-packaging (20.3-1) ... 2022-04-09T11:42:30.505 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2022-04-09T11:42:30.642 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2022-04-09T11:42:30.811 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2022-04-09T11:42:31.183 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2022-04-09T11:42:31.592 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2022-04-09T11:42:32.015 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2022-04-09T11:42:32.442 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-ceph-common (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:42:32.772 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2022-04-09T11:42:32.940 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-html5lib (1.0.1-2) ... 2022-04-09T11:42:33.296 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-04-09T11:42:33.501 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-pluggy (0.13.0-2) ... 2022-04-09T11:42:33.778 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2022-04-09T11:42:33.904 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2022-04-09T11:42:34.051 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2022-04-09T11:42:34.206 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2022-04-09T11:42:34.568 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2022-04-09T11:42:34.859 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-logutils (0.3.3-7) ... 2022-04-09T11:42:35.220 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2022-04-09T11:42:35.341 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2022-04-09T11:42:35.664 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-prettytable (0.7.2-5) ... 2022-04-09T11:42:35.960 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2022-04-09T11:42:36.270 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2022-04-09T11:42:36.364 INFO:teuthology.orchestra.run.smithi119.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2022-04-09T11:42:36.623 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2022-04-09T11:42:36.780 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-04-09T11:42:37.109 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2022-04-09T11:42:37.542 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-04-09T11:42:37.825 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-04-09T11:42:37.972 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-webob (1:1.8.5-2) ... 2022-04-09T11:42:38.285 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-04-09T11:42:38.607 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2022-04-09T11:42:40.191 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-04-09T11:42:40.309 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-pytest (4.6.9-1) ... 2022-04-09T11:42:40.697 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2022-04-09T11:42:41.049 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2022-04-09T11:42:41.421 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2022-04-09T11:42:41.563 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-bs4 (4.8.2-1) ... 2022-04-09T11:42:41.937 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2022-04-09T11:42:42.266 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-04-09T11:42:43.419 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-04-09T11:42:44.686 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-04-09T11:42:44.848 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2022-04-09T11:42:45.025 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2022-04-09T11:42:45.164 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-04-09T11:42:45.507 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2022-04-09T11:42:45.667 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-rsa (4.0-3) ... 2022-04-09T11:42:46.136 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2022-04-09T11:42:46.287 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-webtest (2.0.32-2) ... 2022-04-09T11:42:46.583 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-mgr-modules-core (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:42:46.738 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-04-09T11:42:47.028 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-pastescript (2.0.2-4) ... 2022-04-09T11:42:47.382 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2022-04-09T11:42:47.784 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up librados2 (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:42:47.910 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up librgw2 (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:42:48.034 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libsqlite3-mod-ceph (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:42:48.194 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libcephfs2 (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:42:48.417 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-routes (2.4.1-2) ... 2022-04-09T11:42:48.848 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libradosstriper1 (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:42:48.971 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-google-auth (1.5.1-2) ... 2022-04-09T11:42:49.285 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up librbd1 (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:42:49.453 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-fuse (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:42:49.717 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-04-09T11:42:49.718 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-04-09T11:42:50.664 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libcephfs-dev (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:42:50.797 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-rados (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:42:50.923 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2022-04-09T11:42:52.019 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-rbd (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:42:52.178 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up rbd-fuse (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:42:52.389 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-rgw (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:42:52.645 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-cephfs (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:42:52.771 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-common (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:42:53.358 INFO:teuthology.orchestra.run.smithi119.stdout:Adding group ceph....done 2022-04-09T11:42:54.140 INFO:teuthology.orchestra.run.smithi119.stdout:Adding system user ceph....done 2022-04-09T11:42:54.401 INFO:teuthology.orchestra.run.smithi119.stdout:Setting system user ceph properties....done 2022-04-09T11:42:54.427 INFO:teuthology.orchestra.run.smithi119.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2022-04-09T11:42:54.569 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2022-04-09T11:42:55.085 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2022-04-09T11:42:56.051 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-test (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:42:56.186 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up radosgw (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:42:56.475 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-04-09T11:42:56.476 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-04-09T11:42:57.742 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-base (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:42:58.137 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2022-04-09T11:42:59.138 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-mds (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:42:59.428 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-04-09T11:42:59.428 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-04-09T11:43:00.411 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-mgr (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:43:00.641 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-04-09T11:43:00.642 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-04-09T11:43:01.531 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-osd (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:43:02.293 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-04-09T11:43:02.294 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-04-09T11:43:03.235 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-mgr-k8sevents (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:43:03.398 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-mgr-diskprediction-local (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:43:03.591 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-mon (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:43:03.849 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-04-09T11:43:03.849 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-04-09T11:43:04.754 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-mgr-cephadm (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:43:04.938 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:43:05.098 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-mgr-dashboard (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:43:05.245 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-volume (17.0.0-11488-ga70e525e-1focal) ... 2022-04-09T11:43:05.399 INFO:teuthology.orchestra.run.smithi119.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.7) ... 2022-04-09T11:43:05.591 INFO:teuthology.orchestra.run.smithi119.stdout:Processing triggers for systemd (245.4-4ubuntu3.15) ... 2022-04-09T11:43:06.210 INFO:teuthology.orchestra.run.smithi119.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-04-09T11:43:08.927 INFO:teuthology.orchestra.run.smithi119.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2022-04-09T11:43:09.860 INFO:teuthology.orchestra.run.smithi119.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-04-09T11:43:09.863 DEBUG:teuthology.parallel:result is None 2022-04-09T11:43:09.864 DEBUG:teuthology.orchestra.run.smithi119:> sudo lsb_release -is 2022-04-09T11:43:09.913 INFO:teuthology.orchestra.run.smithi119.stdout:Ubuntu 2022-04-09T11:43:09.919 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-04-09T11:43:09.920 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=a70e525ea98f7194c5e516757c6105ca14c81cb7 2022-04-09T11:43:10.127 DEBUG:teuthology.orchestra.run.smithi119:> dpkg-query -W -f '${Version}' ceph 2022-04-09T11:43:10.150 INFO:teuthology.orchestra.run.smithi119.stdout:17.0.0-11488-ga70e525e-1focal 2022-04-09T11:43:10.151 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-11488-ga70e525e-1focal 2022-04-09T11:43:10.151 INFO:teuthology.task.install:The correct ceph version 17.0.0-11488-ga70e525e-1focal is installed. 2022-04-09T11:43:10.152 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2022-04-09T11:43:10.153 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-04-09T11:43:10.154 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-04-09T11:43:10.206 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2022-04-09T11:43:10.206 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-04-09T11:43:10.207 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/daemon-helper 2022-04-09T11:43:10.218 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-04-09T11:43:10.273 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2022-04-09T11:43:10.274 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-04-09T11:43:10.274 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/adjust-ulimits 2022-04-09T11:43:10.286 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-04-09T11:43:10.297 INFO:teuthology.run_tasks:Running task radosbench... 2022-04-09T11:43:10.316 INFO:tasks.radosbench:Beginning radosbench... 2022-04-09T11:43:10.316 INFO:tasks.rook.ceph_manager.ceph:creating pool_name unique_pool_0 2022-04-09T11:43:10.317 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph osd pool create unique_pool_0 16 2022-04-09T11:43:11.730 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'unique_pool_0' created 2022-04-09T11:43:11.752 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2022-04-09T11:43:12.754 INFO:teuthology.orchestra.run.smithi119.stderr:enabled application 'rados' on pool 'unique_pool_0' 2022-04-09T11:43:13.783 DEBUG:teuthology.orchestra.run.smithi119:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.a -b 65536 --object-size 65536 -p unique_pool_0 bench 360 write' 2022-04-09T11:43:13.786 INFO:teuthology.run_tasks:Running task rook.shell... 2022-04-09T11:43:13.799 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch host label add '`hostname`' 'foo 2022-04-09T11:43:13.800 DEBUG:teuthology.orchestra.run.smithi119:> ceph' orch host ls '|' grep 'foo 2022-04-09T11:43:13.800 DEBUG:teuthology.orchestra.run.smithi119:> ceph' orch host label rm '`hostname`' 'foo 2022-04-09T11:43:13.800 DEBUG:teuthology.orchestra.run.smithi119:> ceph' orch host ls '|' grep -v 'foo 2022-04-09T11:43:13.801 DEBUG:teuthology.orchestra.run.smithi119:> ' 2022-04-09T11:43:13.886 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:hints = 1 2022-04-09T11:43:13.887 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 360 seconds or 0 objects 2022-04-09T11:43:13.887 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:Object prefix: benchmark_data_smithi119_42268 2022-04-09T11:43:13.888 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-04-09T11:43:13.888 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 0 0 0 0 0 0 - 0 2022-04-09T11:43:14.198 INFO:teuthology.orchestra.run.smithi119.stdout:ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) 2022-04-09T11:43:14.221 INFO:teuthology.run_tasks:Running task exec... 2022-04-09T11:43:14.237 INFO:teuthology.task.exec:Executing custom commands... 2022-04-09T11:43:14.238 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi119.front.sepia.ceph.com 2022-04-09T11:43:14.238 DEBUG:teuthology.orchestra.run.smithi119:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'set -ex 2022-04-09T11:43:14.239 DEBUG:teuthology.orchestra.run.smithi119:> toolbox() { 2022-04-09T11:43:14.239 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- "$@" 2022-04-09T11:43:14.239 DEBUG:teuthology.orchestra.run.smithi119:> } 2022-04-09T11:43:14.240 DEBUG:teuthology.orchestra.run.smithi119:> orig_num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-04-09T11:43:14.240 DEBUG:teuthology.orchestra.run.smithi119:> toolbox ceph orch osd rm 0 --force 2022-04-09T11:43:14.240 DEBUG:teuthology.orchestra.run.smithi119:> removed_pv="" 2022-04-09T11:43:14.240 DEBUG:teuthology.orchestra.run.smithi119:> while [ "$removed_pv" = "" ] 2022-04-09T11:43:14.241 DEBUG:teuthology.orchestra.run.smithi119:> do 2022-04-09T11:43:14.241 DEBUG:teuthology.orchestra.run.smithi119:> removed_pv=`kubectl get pv | grep Released | cut -f1 -d " "` 2022-04-09T11:43:14.241 DEBUG:teuthology.orchestra.run.smithi119:> sleep 3s 2022-04-09T11:43:14.242 DEBUG:teuthology.orchestra.run.smithi119:> done 2022-04-09T11:43:14.242 DEBUG:teuthology.orchestra.run.smithi119:> target_path=`kubectl get pv $removed_pv -o jsonpath='"'"'{.spec.local.path}'"'"'` 2022-04-09T11:43:14.242 DEBUG:teuthology.orchestra.run.smithi119:> host=`echo $removed_pv | cut -f1 -d "-"` 2022-04-09T11:43:14.242 DEBUG:teuthology.orchestra.run.smithi119:> toolbox ceph orch device zap $host $target_path --force 2022-04-09T11:43:14.243 DEBUG:teuthology.orchestra.run.smithi119:> zap_completion="0" 2022-04-09T11:43:14.243 DEBUG:teuthology.orchestra.run.smithi119:> while [ "$zap_completion" = "0" ] 2022-04-09T11:43:14.243 DEBUG:teuthology.orchestra.run.smithi119:> do 2022-04-09T11:43:14.244 DEBUG:teuthology.orchestra.run.smithi119:> zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap -o jsonpath='"'"'{.status.succeeded.path}'"'"'` 2022-04-09T11:43:14.244 DEBUG:teuthology.orchestra.run.smithi119:> sleep 3s 2022-04-09T11:43:14.244 DEBUG:teuthology.orchestra.run.smithi119:> done 2022-04-09T11:43:14.244 DEBUG:teuthology.orchestra.run.smithi119:> kubectl patch pv $removed_pv -p '"'"'{"spec":{"claimRef": null}}'"'"' 2022-04-09T11:43:14.245 DEBUG:teuthology.orchestra.run.smithi119:> toolbox ceph orch apply osd --all-available-devices 2022-04-09T11:43:14.245 DEBUG:teuthology.orchestra.run.smithi119:> kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-04-09T11:43:14.245 DEBUG:teuthology.orchestra.run.smithi119:> num_osd="0" 2022-04-09T11:43:14.246 DEBUG:teuthology.orchestra.run.smithi119:> while [ "$num_osd" != "$orig_num_osd" ] 2022-04-09T11:43:14.246 DEBUG:teuthology.orchestra.run.smithi119:> do 2022-04-09T11:43:14.246 DEBUG:teuthology.orchestra.run.smithi119:> echo "waiting for osd to come back up" 2022-04-09T11:43:14.246 DEBUG:teuthology.orchestra.run.smithi119:> num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-04-09T11:43:14.247 DEBUG:teuthology.orchestra.run.smithi119:> sleep 30s 2022-04-09T11:43:14.247 DEBUG:teuthology.orchestra.run.smithi119:> done 2022-04-09T11:43:14.247 DEBUG:teuthology.orchestra.run.smithi119:> ' 2022-04-09T11:43:14.261 INFO:teuthology.orchestra.run.smithi119.stderr:++ toolbox ceph osd stat 2022-04-09T11:43:14.261 INFO:teuthology.orchestra.run.smithi119.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-04-09T11:43:14.261 INFO:teuthology.orchestra.run.smithi119.stderr:++ cut -f3 -d ' ' 2022-04-09T11:43:14.887 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 1 16 1046 1030 64.3731 64.375 0.0144431 0.0154162 2022-04-09T11:43:15.887 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 2 16 2309 2293 71.6519 78.9375 0.0113961 0.013889 2022-04-09T11:43:16.888 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 3 16 3369 3353 69.8493 66.25 0.0224266 0.0140681 2022-04-09T11:43:17.888 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 4 16 4129 4113 64.2609 47.5 0.0101798 0.0155377 2022-04-09T11:43:18.889 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 5 16 5110 5094 63.6619 61.3125 0.00698706 0.015675 2022-04-09T11:43:19.889 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 6 16 6023 6007 62.5612 57.0625 0.0154574 0.0159617 2022-04-09T11:43:20.889 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 7 16 7452 7436 66.3813 89.3125 0.0175961 0.0150482 2022-04-09T11:43:21.889 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 8 16 8742 8726 68.1608 80.625 0.011765 0.0146486 2022-04-09T11:43:21.915 INFO:teuthology.orchestra.run.smithi119.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-04-09T11:43:22.668 INFO:teuthology.orchestra.run.smithi119.stderr:+ orig_num_osd=4 2022-04-09T11:43:22.669 INFO:teuthology.orchestra.run.smithi119.stderr:+ toolbox ceph orch osd rm 0 --force 2022-04-09T11:43:22.669 INFO:teuthology.orchestra.run.smithi119.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch osd rm 0 --force 2022-04-09T11:43:22.744 INFO:teuthology.orchestra.run.smithi119.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-04-09T11:43:22.889 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 9 16 9824 9808 68.1005 67.625 0.0119411 0.0146698 2022-04-09T11:43:23.600 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:43:23.598246 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-04-09T11:43:23.601 INFO:tasks.rook.operator.smithi119.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", AllowUnsupported: true}, 2022-04-09T11:43:23.601 INFO:tasks.rook.operator.smithi119.stdout: Storage: v1.StorageScopeSpec{ 2022-04-09T11:43:23.602 INFO:tasks.rook.operator.smithi119.stdout: ... // 3 identical fields 2022-04-09T11:43:23.602 INFO:tasks.rook.operator.smithi119.stdout: Config: nil, 2022-04-09T11:43:23.603 INFO:tasks.rook.operator.smithi119.stdout: Selection: {}, 2022-04-09T11:43:23.603 INFO:tasks.rook.operator.smithi119.stdout: StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-04-09T11:43:23.604 INFO:tasks.rook.operator.smithi119.stdout:- { 2022-04-09T11:43:23.604 INFO:tasks.rook.operator.smithi119.stdout:- Name: "smithi119-lv-1", 2022-04-09T11:43:23.605 INFO:tasks.rook.operator.smithi119.stdout:- Count: 1, 2022-04-09T11:43:23.605 INFO:tasks.rook.operator.smithi119.stdout:- VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-04-09T11:43:23.606 INFO:tasks.rook.operator.smithi119.stdout:- }, 2022-04-09T11:43:23.606 INFO:tasks.rook.operator.smithi119.stdout: {Name: "smithi119-lv-2", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi119-lv-2", StorageClassName: &"scratch", ...}}}}, 2022-04-09T11:43:23.606 INFO:tasks.rook.operator.smithi119.stdout: {Name: "smithi119-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi119-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-04-09T11:43:23.607 INFO:tasks.rook.operator.smithi119.stdout: {Name: "smithi119-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi119-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-04-09T11:43:23.607 INFO:tasks.rook.operator.smithi119.stdout: }, 2022-04-09T11:43:23.608 INFO:tasks.rook.operator.smithi119.stdout: }, 2022-04-09T11:43:23.608 INFO:tasks.rook.operator.smithi119.stdout: Annotations: nil, 2022-04-09T11:43:23.609 INFO:tasks.rook.operator.smithi119.stdout: Labels: nil, 2022-04-09T11:43:23.609 INFO:tasks.rook.operator.smithi119.stdout: ... // 20 identical fields 2022-04-09T11:43:23.610 INFO:tasks.rook.operator.smithi119.stdout: } 2022-04-09T11:43:23.610 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:43:23.598338 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-04-09T11:43:23.703 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:43:23.704126 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:43:23.747 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:43:23.748869 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7... 2022-04-09T11:43:23.890 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 10 16 10910 10894 68.0773 67.875 0.0199384 0.0146756 2022-04-09T11:43:24.890 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 11 16 12092 12076 68.6037 73.875 0.0164025 0.0145621 2022-04-09T11:43:25.890 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 12 16 12418 12402 64.5846 20.375 0.0276895 0.0144974 2022-04-09T11:43:26.890 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 13 16 12418 12402 59.6167 0 - 0.0144974 2022-04-09T11:43:27.890 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 14 16 12418 12402 55.3585 0 - 0.0144974 2022-04-09T11:43:28.891 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 15 16 12418 12402 51.6681 0 - 0.0144974 2022-04-09T11:43:29.614 INFO:teuthology.orchestra.run.smithi119.stdout:Success 2022-04-09T11:43:29.633 INFO:teuthology.orchestra.run.smithi119.stderr:+ removed_pv= 2022-04-09T11:43:29.633 INFO:teuthology.orchestra.run.smithi119.stderr:+ '[' '' = '' ']' 2022-04-09T11:43:29.634 INFO:teuthology.orchestra.run.smithi119.stderr:++ kubectl get pv 2022-04-09T11:43:29.634 INFO:teuthology.orchestra.run.smithi119.stderr:++ grep Released 2022-04-09T11:43:29.634 INFO:teuthology.orchestra.run.smithi119.stderr:++ cut -f1 -d ' ' 2022-04-09T11:43:29.775 INFO:teuthology.orchestra.run.smithi119.stderr:+ removed_pv= 2022-04-09T11:43:29.775 INFO:teuthology.orchestra.run.smithi119.stderr:+ sleep 3s 2022-04-09T11:43:29.891 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 16 16 12418 12402 48.4389 0 - 0.0144974 2022-04-09T11:43:30.891 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 17 16 12418 12402 45.5896 0 - 0.0144974 2022-04-09T11:43:31.891 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 18 16 12418 12402 43.057 0 - 0.0144974 2022-04-09T11:43:32.777 INFO:teuthology.orchestra.run.smithi119.stderr:+ '[' '' = '' ']' 2022-04-09T11:43:32.777 INFO:teuthology.orchestra.run.smithi119.stderr:++ kubectl get pv 2022-04-09T11:43:32.777 INFO:teuthology.orchestra.run.smithi119.stderr:++ grep Released 2022-04-09T11:43:32.778 INFO:teuthology.orchestra.run.smithi119.stderr:++ cut -f1 -d ' ' 2022-04-09T11:43:32.834 INFO:teuthology.orchestra.run.smithi119.stderr:+ removed_pv= 2022-04-09T11:43:32.834 INFO:teuthology.orchestra.run.smithi119.stderr:+ sleep 3s 2022-04-09T11:43:32.891 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 19 16 12664 12648 41.6 2.19643 0.00236105 0.0219093 2022-04-09T11:43:33.891 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:2022-04-09T11:43:33.892494+0000 min lat: 0.00166103 max lat: 6.82418 avg lat: 0.0219093 2022-04-09T11:43:33.892 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-04-09T11:43:33.892 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 20 16 12664 12648 39.52 0 - 0.0219093 2022-04-09T11:43:34.892 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 21 16 13206 13190 39.251 16.9375 0.01699 0.0254646 2022-04-09T11:43:35.836 INFO:teuthology.orchestra.run.smithi119.stderr:+ '[' '' = '' ']' 2022-04-09T11:43:35.837 INFO:teuthology.orchestra.run.smithi119.stderr:++ kubectl get pv 2022-04-09T11:43:35.837 INFO:teuthology.orchestra.run.smithi119.stderr:++ grep Released 2022-04-09T11:43:35.837 INFO:teuthology.orchestra.run.smithi119.stderr:++ cut -f1 -d ' ' 2022-04-09T11:43:35.892 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 22 16 13937 13921 39.5435 45.6875 0.0131063 0.0252731 2022-04-09T11:43:36.004 INFO:teuthology.orchestra.run.smithi119.stderr:+ removed_pv= 2022-04-09T11:43:36.004 INFO:teuthology.orchestra.run.smithi119.stderr:+ sleep 3s 2022-04-09T11:43:36.892 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 23 16 14710 14694 39.9245 48.3125 0.0100584 0.0250344 2022-04-09T11:43:37.892 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 24 16 15494 15478 40.3024 49 0.0291534 0.0247909 2022-04-09T11:43:38.446 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:43:38.446074 I | ceph-spec: object "rook-ceph-osd-0" matched on delete, reconciling 2022-04-09T11:43:38.892 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 25 16 15947 15931 39.8228 28.3125 0.00362319 0.0245596 2022-04-09T11:43:39.005 INFO:teuthology.orchestra.run.smithi119.stderr:+ '[' '' = '' ']' 2022-04-09T11:43:39.006 INFO:teuthology.orchestra.run.smithi119.stderr:++ kubectl get pv 2022-04-09T11:43:39.006 INFO:teuthology.orchestra.run.smithi119.stderr:++ grep Released 2022-04-09T11:43:39.007 INFO:teuthology.orchestra.run.smithi119.stderr:++ cut -f1 -d ' ' 2022-04-09T11:43:39.064 INFO:teuthology.orchestra.run.smithi119.stderr:+ removed_pv=smithi119-lv-1 2022-04-09T11:43:39.064 INFO:teuthology.orchestra.run.smithi119.stderr:+ sleep 3s 2022-04-09T11:43:39.893 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 26 16 15947 15931 38.2911 0 - 0.0245596 2022-04-09T11:43:40.893 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 27 16 15947 15931 36.8729 0 - 0.0245596 2022-04-09T11:43:41.893 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 28 16 15947 15931 35.5561 0 - 0.0245596 2022-04-09T11:43:42.065 INFO:teuthology.orchestra.run.smithi119.stderr:+ '[' smithi119-lv-1 = '' ']' 2022-04-09T11:43:42.066 INFO:teuthology.orchestra.run.smithi119.stderr:++ kubectl get pv smithi119-lv-1 -o 'jsonpath={.spec.local.path}' 2022-04-09T11:43:42.356 INFO:teuthology.orchestra.run.smithi119.stderr:+ target_path=/dev/vg_nvme/lv_1 2022-04-09T11:43:42.357 INFO:teuthology.orchestra.run.smithi119.stderr:++ echo smithi119-lv-1 2022-04-09T11:43:42.357 INFO:teuthology.orchestra.run.smithi119.stderr:++ cut -f1 -d - 2022-04-09T11:43:42.358 INFO:teuthology.orchestra.run.smithi119.stderr:+ host=smithi119 2022-04-09T11:43:42.358 INFO:teuthology.orchestra.run.smithi119.stderr:+ toolbox ceph orch device zap smithi119 /dev/vg_nvme/lv_1 --force 2022-04-09T11:43:42.358 INFO:teuthology.orchestra.run.smithi119.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch device zap smithi119 /dev/vg_nvme/lv_1 --force 2022-04-09T11:43:42.409 INFO:teuthology.orchestra.run.smithi119.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-04-09T11:43:42.896 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 29 16 15947 15931 34.33 0 - 0.0245596 2022-04-09T11:43:43.200 INFO:teuthology.orchestra.run.smithi119.stdout:/dev/vg_nvme/lv_1 on smithi119 zapped 2022-04-09T11:43:43.223 INFO:teuthology.orchestra.run.smithi119.stderr:+ zap_completion=0 2022-04-09T11:43:43.224 INFO:teuthology.orchestra.run.smithi119.stderr:+ '[' 0 = 0 ']' 2022-04-09T11:43:43.224 INFO:teuthology.orchestra.run.smithi119.stderr:++ kubectl get job -n rook-ceph rook-ceph-device-zap -o 'jsonpath={.status.succeeded.path}' 2022-04-09T11:43:43.435 INFO:teuthology.orchestra.run.smithi119.stderr:+ zap_completion= 2022-04-09T11:43:43.436 INFO:teuthology.orchestra.run.smithi119.stderr:+ sleep 3s 2022-04-09T11:43:43.894 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 30 16 15947 15931 33.1857 0 - 0.0245596 2022-04-09T11:43:44.894 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 31 16 15947 15931 32.1152 0 - 0.0245596 2022-04-09T11:43:45.895 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 32 16 15947 15931 31.1116 0 - 0.0245596 2022-04-09T11:43:46.437 INFO:teuthology.orchestra.run.smithi119.stderr:+ '[' '' = 0 ']' 2022-04-09T11:43:46.437 INFO:teuthology.orchestra.run.smithi119.stderr:+ kubectl patch pv smithi119-lv-1 -p '{"spec":{"claimRef": null}}' 2022-04-09T11:43:46.504 INFO:teuthology.orchestra.run.smithi119.stdout:persistentvolume/smithi119-lv-1 patched 2022-04-09T11:43:46.507 INFO:teuthology.orchestra.run.smithi119.stderr:+ toolbox ceph orch apply osd --all-available-devices 2022-04-09T11:43:46.507 INFO:teuthology.orchestra.run.smithi119.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch apply osd --all-available-devices 2022-04-09T11:43:46.607 INFO:teuthology.orchestra.run.smithi119.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-04-09T11:43:46.895 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 33 16 15947 15931 30.1689 0 - 0.0245596 2022-04-09T11:43:47.895 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 34 16 15947 15931 29.2816 0 - 0.0245596 2022-04-09T11:43:48.103 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:43:48.098529 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-04-09T11:43:48.103 INFO:tasks.rook.operator.smithi119.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", AllowUnsupported: true}, 2022-04-09T11:43:48.103 INFO:tasks.rook.operator.smithi119.stdout: Storage: v1.StorageScopeSpec{ 2022-04-09T11:43:48.104 INFO:tasks.rook.operator.smithi119.stdout: ... // 3 identical fields 2022-04-09T11:43:48.104 INFO:tasks.rook.operator.smithi119.stdout: Config: nil, 2022-04-09T11:43:48.104 INFO:tasks.rook.operator.smithi119.stdout: Selection: {}, 2022-04-09T11:43:48.104 INFO:tasks.rook.operator.smithi119.stdout: StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-04-09T11:43:48.106 INFO:tasks.rook.operator.smithi119.stdout: {Name: "smithi119-lv-2", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi119-lv-2", StorageClassName: &"scratch", ...}}}}, 2022-04-09T11:43:48.106 INFO:tasks.rook.operator.smithi119.stdout: {Name: "smithi119-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi119-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-04-09T11:43:48.106 INFO:tasks.rook.operator.smithi119.stdout: {Name: "smithi119-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi119-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-04-09T11:43:48.106 INFO:tasks.rook.operator.smithi119.stdout:+ { 2022-04-09T11:43:48.107 INFO:tasks.rook.operator.smithi119.stdout:+ Name: "smithi119-lv-1", 2022-04-09T11:43:48.107 INFO:tasks.rook.operator.smithi119.stdout:+ Count: 1, 2022-04-09T11:43:48.107 INFO:tasks.rook.operator.smithi119.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-04-09T11:43:48.107 INFO:tasks.rook.operator.smithi119.stdout:+ }, 2022-04-09T11:43:48.108 INFO:tasks.rook.operator.smithi119.stdout: }, 2022-04-09T11:43:48.108 INFO:tasks.rook.operator.smithi119.stdout: }, 2022-04-09T11:43:48.108 INFO:tasks.rook.operator.smithi119.stdout: Annotations: nil, 2022-04-09T11:43:48.108 INFO:tasks.rook.operator.smithi119.stdout: Labels: nil, 2022-04-09T11:43:48.108 INFO:tasks.rook.operator.smithi119.stdout: ... // 20 identical fields 2022-04-09T11:43:48.109 INFO:tasks.rook.operator.smithi119.stdout: } 2022-04-09T11:43:48.109 INFO:teuthology.orchestra.run.smithi119.stdout:['Success'] 2022-04-09T11:43:48.133 INFO:teuthology.orchestra.run.smithi119.stderr:+ kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-04-09T11:43:48.743 INFO:teuthology.orchestra.run.smithi119.stdout:job.batch "rook-ceph-device-zap" deleted 2022-04-09T11:43:48.895 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 35 16 16140 16124 28.7895 1.20625 0.00256032 0.0338882 2022-04-09T11:43:49.095 INFO:teuthology.orchestra.run.smithi119.stderr:+ num_osd=0 2022-04-09T11:43:49.095 INFO:teuthology.orchestra.run.smithi119.stderr:+ '[' 0 '!=' 4 ']' 2022-04-09T11:43:49.096 INFO:teuthology.orchestra.run.smithi119.stderr:+ echo 'waiting for osd to come back up' 2022-04-09T11:43:49.096 INFO:teuthology.orchestra.run.smithi119.stdout:waiting for osd to come back up 2022-04-09T11:43:49.097 INFO:teuthology.orchestra.run.smithi119.stderr:++ toolbox ceph osd stat 2022-04-09T11:43:49.097 INFO:teuthology.orchestra.run.smithi119.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-04-09T11:43:49.097 INFO:teuthology.orchestra.run.smithi119.stderr:++ cut -f3 -d ' ' 2022-04-09T11:43:49.300 INFO:teuthology.orchestra.run.smithi119.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-04-09T11:43:49.895 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 36 16 16140 16124 27.9899 0 - 0.0338882 2022-04-09T11:43:49.917 INFO:teuthology.orchestra.run.smithi119.stderr:+ num_osd=3 2022-04-09T11:43:49.918 INFO:teuthology.orchestra.run.smithi119.stderr:+ sleep 30s 2022-04-09T11:43:50.037 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:43:50.029110 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11488 quincy" 2022-04-09T11:43:50.037 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:43:50.029133 I | ceph-cluster-controller: validating ceph version from provided image 2022-04-09T11:43:50.038 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:43:50.029143 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11488 quincy", pursuing 2022-04-09T11:43:50.244 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:43:50.240704 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:43:50.895 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 37 16 16140 16124 27.2334 0 - 0.0338882 2022-04-09T11:43:51.058 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:43:50.994781 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-09T11:43:51.059 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:43:51.055948 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-09T11:43:51.896 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 38 16 16140 16124 26.5167 0 - 0.0338882 2022-04-09T11:43:52.117 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:43:52.115476 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11488 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7" 2022-04-09T11:43:52.896 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 39 16 16140 16124 25.8368 0 - 0.0338882 2022-04-09T11:43:53.896 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:2022-04-09T11:43:53.894495+0000 min lat: 0.00166103 max lat: 9.73842 avg lat: 0.0338882 2022-04-09T11:43:53.897 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-04-09T11:43:53.897 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 40 16 16140 16124 25.1909 0 - 0.0338882 2022-04-09T11:43:53.911 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:43:53.908492 I | op-mon: start running mons 2022-04-09T11:43:53.915 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:43:53.912895 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:43:54.896 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 41 16 16140 16124 24.5765 0 - 0.0338882 2022-04-09T11:43:55.185 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:43:55.182230 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.24.44:6789"]}] data:a=10.107.24.44:6789 mapping:{"node":{"a":{"Name":"smithi119","Hostname":"smithi119","Address":"172.21.15.119"}}} maxMonId:0] 2022-04-09T11:43:55.897 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 42 16 16180 16164 24.0509 0.357143 0.00267526 0.0381446 2022-04-09T11:43:56.897 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 43 16 16777 16761 24.3592 37.3125 0.00380432 0.0409883 2022-04-09T11:43:57.248 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:43:57.235580 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-09T11:43:57.260 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:43:57.256441 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-09T11:43:57.810 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:43:57.807207 I | op-mon: targeting the mon count 1 2022-04-09T11:43:57.829 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:43:57.826281 I | ceph-cluster-controller: Cluster update requested. CANCELLING CURRENT ORCHESTRATION 2022-04-09T11:43:57.829 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:43:57.826375 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-04-09T11:43:57.897 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 44 16 17632 17616 25.02 53.4375 0.0103178 0.0399555 2022-04-09T11:43:58.010 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:43:58.002970 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:43:58.025 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:43:58.020834 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7... 2022-04-09T11:43:58.060 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:43:58.055096 I | op-k8sutil: Removing previous job rook-ceph-detect-version to start a new one 2022-04-09T11:43:58.065 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:43:58.060721 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-04-09T11:43:58.897 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 45 16 18473 18457 25.6319 52.5625 0.0142461 0.0390001 2022-04-09T11:43:59.898 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 46 16 19227 19211 26.099 47.125 0.0234965 0.0383044 2022-04-09T11:44:00.294 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:00.290759 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-04-09T11:44:00.898 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 47 16 19461 19445 25.8549 14.625 0.0185031 0.0381019 2022-04-09T11:44:01.898 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 48 16 19904 19888 25.893 27.6875 0.00962512 0.0386091 2022-04-09T11:44:02.602 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:02.588428 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-04-09T11:44:02.898 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 49 16 20548 20532 26.1859 40.25 0.0645885 0.038178 2022-04-09T11:44:03.899 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 50 16 21134 21118 26.3944 36.625 0.0178905 0.0378767 2022-04-09T11:44:04.765 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:04.756322 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-04-09T11:44:04.899 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 51 16 21815 21799 26.7114 42.5625 0.0273768 0.0374233 2022-04-09T11:44:05.899 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 52 16 22568 22552 27.1026 47.0625 0.0274699 0.0368852 2022-04-09T11:44:06.764 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:06.760603 I | op-k8sutil: batch job rook-ceph-detect-version deleted 2022-04-09T11:44:06.899 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 53 16 23271 23255 27.4202 43.9375 0.0279768 0.036458 2022-04-09T11:44:07.899 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 54 16 23979 23963 27.7318 44.25 0.0196409 0.036022 2022-04-09T11:44:08.900 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 55 16 24176 24160 27.4514 12.3125 0.020507 0.0364169 2022-04-09T11:44:09.900 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 56 16 24812 24796 27.6709 39.75 0.0216398 0.0361268 2022-04-09T11:44:10.900 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 57 16 25130 25114 27.5341 19.875 0.022047 0.0363071 2022-04-09T11:44:11.900 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 58 16 25837 25821 27.8212 44.1875 0.0151711 0.0359322 2022-04-09T11:44:12.900 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 59 16 26609 26593 28.1674 48.25 0.0116462 0.0354896 2022-04-09T11:44:13.900 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:2022-04-09T11:44:13.896762+0000 min lat: 0.00166103 max lat: 9.73842 avg lat: 0.0353255 2022-04-09T11:44:13.901 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-04-09T11:44:13.901 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 60 16 27186 27170 28.2989 36.0625 0.0332224 0.0353255 2022-04-09T11:44:14.901 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 61 16 27925 27909 28.5921 46.1875 0.0182171 0.0349673 2022-04-09T11:44:15.901 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 62 16 28386 28370 28.5956 28.8125 0.0252542 0.0349056 2022-04-09T11:44:16.901 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 63 16 28923 28907 28.6744 33.5625 0.0185161 0.0348594 2022-04-09T11:44:17.902 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 64 16 29534 29518 28.8225 38.1875 0.0243654 0.0346838 2022-04-09T11:44:18.902 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 65 16 30255 30239 29.0723 45.0625 0.0114335 0.0343863 2022-04-09T11:44:19.903 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 66 16 30654 30638 29.0096 24.9375 0.0154674 0.0344553 2022-04-09T11:44:19.921 INFO:teuthology.orchestra.run.smithi119.stderr:+ '[' 3 '!=' 4 ']' 2022-04-09T11:44:19.922 INFO:teuthology.orchestra.run.smithi119.stderr:+ echo 'waiting for osd to come back up' 2022-04-09T11:44:19.922 INFO:teuthology.orchestra.run.smithi119.stdout:waiting for osd to come back up 2022-04-09T11:44:19.923 INFO:teuthology.orchestra.run.smithi119.stderr:++ toolbox ceph osd stat 2022-04-09T11:44:19.923 INFO:teuthology.orchestra.run.smithi119.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-04-09T11:44:19.924 INFO:teuthology.orchestra.run.smithi119.stderr:++ cut -f3 -d ' ' 2022-04-09T11:44:20.823 INFO:teuthology.orchestra.run.smithi119.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-04-09T11:44:20.903 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 67 16 31152 31136 29.0411 31.125 0.0405526 0.0344237 2022-04-09T11:44:21.903 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 68 16 31647 31631 29.069 30.9375 0.025825 0.0343843 2022-04-09T11:44:22.623 INFO:teuthology.orchestra.run.smithi119.stderr:+ num_osd=3 2022-04-09T11:44:22.623 INFO:teuthology.orchestra.run.smithi119.stderr:+ sleep 30s 2022-04-09T11:44:22.903 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 69 16 32127 32111 29.0824 30 0.0219926 0.0343764 2022-04-09T11:44:23.903 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 70 16 32623 32607 29.1098 31 0.0222114 0.0342731 2022-04-09T11:44:24.903 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 71 16 33267 33251 29.2666 40.25 0.0322693 0.0341563 2022-04-09T11:44:25.903 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 72 15 34013 33998 29.5085 46.6875 0.0438052 0.0338778 2022-04-09T11:44:26.904 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 73 16 34438 34422 29.4672 26.5 0.0156774 0.033923 2022-04-09T11:44:27.905 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 74 16 34941 34925 29.4933 31.4375 0.0271177 0.0338959 2022-04-09T11:44:28.905 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 75 16 35493 35477 29.56 34.5 0.0194747 0.0338199 2022-04-09T11:44:29.905 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 76 16 36147 36131 29.7088 40.875 0.0281115 0.0336514 2022-04-09T11:44:30.905 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 77 16 36829 36813 29.8765 42.625 0.0163351 0.0334624 2022-04-09T11:44:31.906 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 78 16 37277 37261 29.8524 28 0.0329089 0.0334859 2022-04-09T11:44:32.906 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 79 16 37859 37843 29.935 36.375 0.0374225 0.0333678 2022-04-09T11:44:33.906 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:2022-04-09T11:44:33.900976+0000 min lat: 0.00166103 max lat: 9.73842 avg lat: 0.0333389 2022-04-09T11:44:33.906 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-04-09T11:44:33.907 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 80 16 38403 38387 29.9857 34 0.0212155 0.0333389 2022-04-09T11:44:34.906 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 81 16 39006 38990 30.0808 37.6875 0.0242234 0.0332339 2022-04-09T11:44:35.906 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 82 16 39584 39568 30.1544 36.125 0.0309237 0.0331403 2022-04-09T11:44:36.906 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 83 16 40199 40183 30.2542 38.4375 0.0171487 0.0330438 2022-04-09T11:44:37.907 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 84 16 40802 40786 30.3426 37.6875 0.08657 0.0329241 2022-04-09T11:44:38.907 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 85 16 41170 41154 30.2562 23 0.0244856 0.0328857 2022-04-09T11:44:39.907 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 86 16 41596 41580 30.2139 26.625 0.0498259 0.033087 2022-04-09T11:44:40.907 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 87 16 42167 42151 30.2768 35.6875 0.0280717 0.0330182 2022-04-09T11:44:41.907 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 88 16 42745 42729 30.3432 36.125 0.025721 0.032947 2022-04-09T11:44:42.907 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 89 16 43323 43307 30.4081 36.125 0.0270367 0.0328762 2022-04-09T11:44:43.907 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 90 16 43867 43851 30.448 34 0.0140155 0.0328318 2022-04-09T11:44:44.907 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 91 16 44425 44409 30.4966 34.875 0.0194723 0.0327755 2022-04-09T11:44:45.907 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 92 16 45007 44991 30.5605 36.375 0.0210554 0.0327112 2022-04-09T11:44:46.907 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 93 16 45537 45521 30.588 33.125 0.025309 0.0326833 2022-04-09T11:44:47.908 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 94 16 46184 46168 30.6928 40.4375 0.0265072 0.032571 2022-04-09T11:44:48.908 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 95 16 46821 46805 30.7887 39.8125 0.0192969 0.0324705 2022-04-09T11:44:49.908 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 96 16 47299 47283 30.7792 29.875 0.0134351 0.032481 2022-04-09T11:44:50.908 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 97 16 47930 47914 30.8682 39.4375 0.00553724 0.0323722 2022-04-09T11:44:51.908 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 98 16 48099 48083 30.661 10.5625 0.00436771 0.0322934 2022-04-09T11:44:52.626 INFO:teuthology.orchestra.run.smithi119.stderr:+ '[' 3 '!=' 4 ']' 2022-04-09T11:44:52.626 INFO:teuthology.orchestra.run.smithi119.stderr:+ echo 'waiting for osd to come back up' 2022-04-09T11:44:52.627 INFO:teuthology.orchestra.run.smithi119.stdout:waiting for osd to come back up 2022-04-09T11:44:52.628 INFO:teuthology.orchestra.run.smithi119.stderr:++ toolbox ceph osd stat 2022-04-09T11:44:52.628 INFO:teuthology.orchestra.run.smithi119.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-04-09T11:44:52.628 INFO:teuthology.orchestra.run.smithi119.stderr:++ cut -f3 -d ' ' 2022-04-09T11:44:52.744 INFO:teuthology.orchestra.run.smithi119.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-04-09T11:44:52.909 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 99 16 48099 48083 30.3513 0 - 0.0322934 2022-04-09T11:44:53.599 INFO:teuthology.orchestra.run.smithi119.stderr:+ num_osd=3 2022-04-09T11:44:53.599 INFO:teuthology.orchestra.run.smithi119.stderr:+ sleep 30s 2022-04-09T11:44:53.901 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:53.894942 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11488 quincy" 2022-04-09T11:44:53.902 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:53.895064 I | ceph-cluster-controller: validating ceph version from provided image 2022-04-09T11:44:53.902 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:53.895125 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11488 quincy", pursuing 2022-04-09T11:44:53.908 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:2022-04-09T11:44:53.903558+0000 min lat: 0.00166103 max lat: 9.73842 avg lat: 0.0322934 2022-04-09T11:44:53.909 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-04-09T11:44:53.909 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 100 16 48099 48083 30.0478 0 - 0.0322934 2022-04-09T11:44:54.545 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:54.536340 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:44:54.909 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 101 16 48099 48083 29.7503 0 - 0.0322934 2022-04-09T11:44:54.986 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:54.894763 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-09T11:44:54.986 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:54.896426 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-09T11:44:55.288 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:55.282651 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11488 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7" 2022-04-09T11:44:55.604 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:55.598475 I | op-mon: start running mons 2022-04-09T11:44:55.629 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:55.623841 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:44:55.644 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:55.638674 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.24.44:6789"]}] data:a=10.107.24.44:6789 mapping:{"node":{"a":{"Name":"smithi119","Hostname":"smithi119","Address":"172.21.15.119"}}} maxMonId:0] 2022-04-09T11:44:55.650 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:55.644490 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-09T11:44:55.650 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:55.644612 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-09T11:44:55.909 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 102 16 48099 48083 29.4586 0 - 0.0322934 2022-04-09T11:44:56.592 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:56.586901 I | op-mon: targeting the mon count 1 2022-04-09T11:44:56.881 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:56.875708 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-04-09T11:44:56.909 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 103 16 48099 48083 29.1727 0 - 0.0322934 2022-04-09T11:44:57.272 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:57.266078 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-04-09T11:44:57.272 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:57.266098 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-04-09T11:44:57.601 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:57.595484 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-04-09T11:44:57.601 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:57.595511 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-04-09T11:44:57.909 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 104 16 48099 48083 28.8922 0 - 0.0322934 2022-04-09T11:44:57.972 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:57.966835 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-04-09T11:44:57.973 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:57.966871 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-04-09T11:44:58.326 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:58.320912 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-04-09T11:44:58.327 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:58.320947 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-04-09T11:44:58.666 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:58.660361 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-04-09T11:44:58.666 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:58.660389 I | op-config: deleting "log file" option from the mon configuration database 2022-04-09T11:44:58.909 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 105 16 48099 48083 28.617 0 - 0.0322934 2022-04-09T11:44:59.023 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:59.017220 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-04-09T11:44:59.023 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:59.017242 I | op-mon: checking for basic quorum with existing mons 2022-04-09T11:44:59.294 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:59.288497 I | op-mon: mon "a" endpoint is [v2:10.107.24.44:3300,v1:10.107.24.44:6789] 2022-04-09T11:44:59.320 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:59.314421 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.24.44:6789"]}] data:a=10.107.24.44:6789 mapping:{"node":{"a":{"Name":"smithi119","Hostname":"smithi119","Address":"172.21.15.119"}}} maxMonId:0] 2022-04-09T11:44:59.390 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:59.384415 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-09T11:44:59.393 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:59.384698 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-09T11:44:59.436 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:59.430876 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-04-09T11:44:59.443 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:59.438301 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-04-09T11:44:59.444 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:59.438326 I | op-mon: waiting for mon quorum with [a] 2022-04-09T11:44:59.631 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:44:59.625566 I | op-mon: mons running: [a] 2022-04-09T11:44:59.909 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 106 16 48099 48083 28.347 0 - 0.0322934 2022-04-09T11:45:00.195 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:00.183593 I | op-mon: Monitors in quorum: [a] 2022-04-09T11:45:00.196 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:00.183617 I | op-mon: mons created: 1 2022-04-09T11:45:00.909 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 107 16 48569 48553 28.3566 3.26389 0.0465594 0.0352538 2022-04-09T11:45:01.004 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:00.998291 I | op-mon: waiting for mon quorum with [a] 2022-04-09T11:45:01.020 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:01.014306 I | op-mon: mons running: [a] 2022-04-09T11:45:01.858 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:01.852931 I | op-mon: Monitors in quorum: [a] 2022-04-09T11:45:01.859 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:01.852966 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-04-09T11:45:01.909 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 108 16 49099 49083 28.4007 33.125 0.0351705 0.0351994 2022-04-09T11:45:02.792 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:02.786243 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-04-09T11:45:02.909 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 109 16 49589 49573 28.4211 30.625 0.0308365 0.0351753 2022-04-09T11:45:03.584 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:03.577907 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-04-09T11:45:03.909 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 110 16 50036 50020 28.4167 27.9375 0.021456 0.0351791 2022-04-09T11:45:04.381 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:04.374832 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-04-09T11:45:04.909 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 111 16 50583 50567 28.4686 34.1875 0.0510434 0.0351161 2022-04-09T11:45:05.163 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:05.156618 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-04-09T11:45:05.163 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:05.156635 I | cephclient: getting or creating ceph auth key "client.crash" 2022-04-09T11:45:05.910 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 112 16 51060 51044 28.4806 29.8125 0.021372 0.0350524 2022-04-09T11:45:06.180 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:06.174362 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-04-09T11:45:06.910 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 113 16 51495 51479 28.4691 27.1875 0.0156696 0.0351147 2022-04-09T11:45:06.999 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:06.987071 I | cephclient: successfully enabled msgr2 protocol 2022-04-09T11:45:07.000 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:06.987100 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-04-09T11:45:07.000 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:06.987104 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-04-09T11:45:07.764 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:07.757823 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-04-09T11:45:07.802 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:07.796672 I | op-mgr: start running mgr 2022-04-09T11:45:07.803 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:07.796696 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-04-09T11:45:07.910 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 114 16 52060 52044 28.5291 35.3125 0.0196949 0.0350412 2022-04-09T11:45:08.698 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:08.691288 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-04-09T11:45:08.703 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:08.697798 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-04-09T11:45:08.711 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:08.704325 I | op-mgr: setting services to point to mgr "a" 2022-04-09T11:45:08.864 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:08.858921 I | op-osd: start running osds in namespace "rook-ceph" 2022-04-09T11:45:08.865 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:08.859143 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-04-09T11:45:08.878 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:08.873201 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-04-09T11:45:08.881 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:08.875777 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi119-lv-2" 2022-04-09T11:45:08.882 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:08.875800 I | op-osd: OSD PVC "smithi119-lv-2-data-0d2cw5" already exists 2022-04-09T11:45:08.882 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:08.875811 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi119-lv-3" 2022-04-09T11:45:08.882 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:08.875821 I | op-osd: OSD PVC "smithi119-lv-3-data-05666c" already exists 2022-04-09T11:45:08.882 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:08.876027 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi119-lv-4" 2022-04-09T11:45:08.883 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:08.876042 I | op-osd: OSD PVC "smithi119-lv-4-data-0ts767" already exists 2022-04-09T11:45:08.883 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:08.876049 I | op-osd: creating 1 new PVCs for device set "smithi119-lv-1" 2022-04-09T11:45:08.888 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:08.882274 I | op-osd: successfully provisioned PVC "smithi119-lv-1-data-0vcsdm" 2022-04-09T11:45:08.910 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 115 16 52586 52570 28.5669 32.875 0.030265 0.0349873 2022-04-09T11:45:08.952 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:08.946932 I | op-osd: OSD will have its main bluestore block on "smithi119-lv-2-data-0d2cw5" 2022-04-09T11:45:08.953 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:08.946948 I | op-osd: OSD will have its main bluestore block on "smithi119-lv-3-data-05666c" 2022-04-09T11:45:08.953 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:08.946952 I | op-osd: OSD will have its main bluestore block on "smithi119-lv-4-data-0ts767" 2022-04-09T11:45:08.953 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:08.946955 I | op-osd: OSD will have its main bluestore block on "smithi119-lv-1-data-0vcsdm" 2022-04-09T11:45:09.028 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:09.021539 I | op-osd: started OSD provisioning job for PVC "smithi119-lv-1-data-0vcsdm" 2022-04-09T11:45:09.028 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:09.021564 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-04-09T11:45:09.028 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:09.021568 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-04-09T11:45:09.091 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:09.086156 I | op-osd: OSD orchestration status for PVC smithi119-lv-1-data-0vcsdm is "starting" 2022-04-09T11:45:09.910 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 116 16 52959 52943 28.5216 23.3125 0.0350678 0.0350495 2022-04-09T11:45:10.904 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:10.895202 I | op-mgr: successful modules: prometheus 2022-04-09T11:45:10.910 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 117 16 53498 53482 28.5657 33.6875 0.0318277 0.0349945 2022-04-09T11:45:10.911 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:10.900194 I | op-mgr: successful modules: mgr module(s) from the spec 2022-04-09T11:45:10.916 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:10.907533 I | op-mgr: successful modules: dashboard 2022-04-09T11:45:11.371 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:11.365012 I | op-mgr: successful modules: balancer 2022-04-09T11:45:11.498 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:11.491721 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-04-09T11:45:11.503 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:11.497759 I | op-osd: updating OSD 1 on PVC "smithi119-lv-4-data-0ts767" 2022-04-09T11:45:11.503 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:11.497785 I | op-osd: OSD will have its main bluestore block on "smithi119-lv-4-data-0ts767" 2022-04-09T11:45:11.910 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 118 16 53920 53904 28.5471 26.375 0.0262053 0.035018 2022-04-09T11:45:12.646 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:12.640666 I | op-mgr: successful modules: orchestrator modules 2022-04-09T11:45:12.910 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 119 15 54348 54333 28.5325 26.8125 0.0241518 0.0350386 2022-04-09T11:45:13.725 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:13.719539 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-04-09T11:45:13.735 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:13.726798 I | op-osd: updating OSD 2 on PVC "smithi119-lv-2-data-0d2cw5" 2022-04-09T11:45:13.735 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:13.726827 I | op-osd: OSD will have its main bluestore block on "smithi119-lv-2-data-0d2cw5" 2022-04-09T11:45:13.911 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:2022-04-09T11:45:13.905572+0000 min lat: 0.00166103 max lat: 9.73842 avg lat: 0.0352017 2022-04-09T11:45:13.911 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-04-09T11:45:13.911 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 120 16 54555 54539 28.402 12.875 0.0188441 0.0352017 2022-04-09T11:45:14.911 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 121 16 54770 54754 28.2784 13.4375 0.0254074 0.0353543 2022-04-09T11:45:15.818 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:15.811720 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-04-09T11:45:15.824 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:15.816769 I | op-osd: updating OSD 3 on PVC "smithi119-lv-3-data-05666c" 2022-04-09T11:45:15.824 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:15.816929 I | op-osd: OSD will have its main bluestore block on "smithi119-lv-3-data-05666c" 2022-04-09T11:45:15.910 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 122 16 55283 55267 28.3093 32.0625 0.0487872 0.0353171 2022-04-09T11:45:16.911 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 123 16 55795 55779 28.3393 32 0.0216923 0.035278 2022-04-09T11:45:17.911 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 124 16 56043 56027 28.2358 15.5 0.0476749 0.0354074 2022-04-09T11:45:18.911 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 125 15 56565 56550 28.2714 32.6875 0.0281637 0.0353636 2022-04-09T11:45:19.911 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 126 16 57140 57124 28.3317 35.875 0.0183893 0.035288 2022-04-09T11:45:20.911 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 127 16 57634 57618 28.3517 30.875 0.0656753 0.0352616 2022-04-09T11:45:21.911 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 128 16 58165 58149 28.3894 33.1875 0.0314348 0.0352159 2022-04-09T11:45:22.911 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 129 16 58728 58712 28.4421 35.1875 0.0128978 0.0351498 2022-04-09T11:45:23.599 INFO:teuthology.orchestra.run.smithi119.stderr:+ '[' 3 '!=' 4 ']' 2022-04-09T11:45:23.600 INFO:teuthology.orchestra.run.smithi119.stdout:waiting for osd to come back up 2022-04-09T11:45:23.600 INFO:teuthology.orchestra.run.smithi119.stderr:+ echo 'waiting for osd to come back up' 2022-04-09T11:45:23.601 INFO:teuthology.orchestra.run.smithi119.stderr:++ toolbox ceph osd stat 2022-04-09T11:45:23.601 INFO:teuthology.orchestra.run.smithi119.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-04-09T11:45:23.601 INFO:teuthology.orchestra.run.smithi119.stderr:++ cut -f3 -d ' ' 2022-04-09T11:45:23.720 INFO:teuthology.orchestra.run.smithi119.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-04-09T11:45:23.911 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 130 16 59200 59184 28.4502 29.5 0.023172 0.0351384 2022-04-09T11:45:24.307 INFO:teuthology.orchestra.run.smithi119.stderr:+ num_osd=3 2022-04-09T11:45:24.308 INFO:teuthology.orchestra.run.smithi119.stderr:+ sleep 30s 2022-04-09T11:45:24.911 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 131 16 59290 59274 28.276 5.625 0.0425396 0.0351384 2022-04-09T11:45:25.911 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 132 16 59593 59577 28.2052 18.9375 0.023267 0.0354447 2022-04-09T11:45:26.911 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 133 16 60089 60073 28.2262 31 0.0240689 0.0354212 2022-04-09T11:45:27.912 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 134 16 60341 60325 28.1331 15.75 0.0313748 0.0354177 2022-04-09T11:45:28.912 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 135 16 60341 60325 27.9247 0 - 0.0354177 2022-04-09T11:45:29.912 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 136 16 60737 60721 27.9013 12.375 0.0169216 0.0358128 2022-04-09T11:45:30.912 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 137 16 60737 60721 27.6977 0 - 0.0358128 2022-04-09T11:45:31.912 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 138 16 60928 60912 27.5835 5.96875 0.04774 0.0362436 2022-04-09T11:45:32.912 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 139 16 61454 61438 27.6215 32.875 0.0152361 0.0361939 2022-04-09T11:45:33.912 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:2022-04-09T11:45:33.907630+0000 min lat: 0.00166103 max lat: 9.73842 avg lat: 0.0361819 2022-04-09T11:45:33.913 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-04-09T11:45:33.913 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 140 16 61921 61905 27.6327 29.1875 0.031545 0.0361819 2022-04-09T11:45:34.912 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 141 16 62367 62351 27.6344 27.875 0.0216007 0.0361631 2022-04-09T11:45:35.913 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 142 16 62897 62881 27.673 33.125 0.0462919 0.0361249 2022-04-09T11:45:36.912 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 143 16 63357 63341 27.6805 28.75 0.0415762 0.0361158 2022-04-09T11:45:37.913 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 144 16 63814 63798 27.6866 28.5625 0.0121774 0.0361078 2022-04-09T11:45:38.913 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 145 16 64338 64322 27.7215 32.75 0.0168589 0.0360657 2022-04-09T11:45:39.913 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 146 16 64925 64909 27.7829 36.6875 0.0251853 0.0359846 2022-04-09T11:45:40.913 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 147 16 65390 65374 27.7916 29.0625 0.0327239 0.0359729 2022-04-09T11:45:41.913 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 148 16 65806 65790 27.7795 26 0.0893448 0.0359853 2022-04-09T11:45:42.913 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 149 16 66245 66229 27.7772 27.4375 0.0206059 0.0359915 2022-04-09T11:45:43.913 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 150 16 66740 66724 27.7982 30.9375 0.0320632 0.0359646 2022-04-09T11:45:44.913 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 151 16 67154 67138 27.7855 25.875 0.0319689 0.0359799 2022-04-09T11:45:45.913 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 152 16 67690 67674 27.823 33.5 0.027339 0.035927 2022-04-09T11:45:46.913 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 153 16 68192 68176 27.8462 31.375 0.0675209 0.0359007 2022-04-09T11:45:47.914 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 154 16 68670 68654 27.8594 29.875 0.0284436 0.0358706 2022-04-09T11:45:48.914 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 155 16 69180 69164 27.8853 31.875 0.0316148 0.0358516 2022-04-09T11:45:49.914 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 156 16 69663 69647 27.9 30.1875 0.0267521 0.0358322 2022-04-09T11:45:50.914 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 157 16 70215 70199 27.942 34.5 0.0237104 0.0357798 2022-04-09T11:45:51.914 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 158 16 70681 70665 27.9495 29.125 0.0478929 0.03577 2022-04-09T11:45:52.914 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 159 16 71146 71130 27.9565 29.0625 0.0248822 0.0357606 2022-04-09T11:45:53.406 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:53.396674 I | op-osd: OSD orchestration status for node smithi119-lv-1-data-0vcsdm is "orchestrating" 2022-04-09T11:45:53.914 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:2022-04-09T11:45:53.909608+0000 min lat: 0.00166103 max lat: 9.73842 avg lat: 0.0357652 2022-04-09T11:45:53.914 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-04-09T11:45:53.915 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 160 16 71585 71569 27.9532 27.4375 0.058801 0.0357652 2022-04-09T11:45:54.149 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:45:54.143190 I | op-osd: OSD orchestration status for PVC smithi119-lv-1-data-0vcsdm is "orchestrating" 2022-04-09T11:45:54.308 INFO:teuthology.orchestra.run.smithi119.stderr:+ '[' 3 '!=' 4 ']' 2022-04-09T11:45:54.309 INFO:teuthology.orchestra.run.smithi119.stderr:+ echo 'waiting for osd to come back up' 2022-04-09T11:45:54.309 INFO:teuthology.orchestra.run.smithi119.stdout:waiting for osd to come back up 2022-04-09T11:45:54.310 INFO:teuthology.orchestra.run.smithi119.stderr:++ toolbox ceph osd stat 2022-04-09T11:45:54.310 INFO:teuthology.orchestra.run.smithi119.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-04-09T11:45:54.310 INFO:teuthology.orchestra.run.smithi119.stderr:++ cut -f3 -d ' ' 2022-04-09T11:45:54.398 INFO:teuthology.orchestra.run.smithi119.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-04-09T11:45:54.914 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 161 16 71994 71978 27.9383 25.5625 0.0370791 0.0357833 2022-04-09T11:45:55.142 INFO:teuthology.orchestra.run.smithi119.stderr:+ num_osd=3 2022-04-09T11:45:55.142 INFO:teuthology.orchestra.run.smithi119.stderr:+ sleep 30s 2022-04-09T11:45:55.915 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 162 16 72453 72437 27.9428 28.6875 0.0399704 0.0357775 2022-04-09T11:45:56.916 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 163 16 72903 72887 27.9439 28.125 0.036356 0.035776 2022-04-09T11:45:57.915 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 164 16 73297 73281 27.9237 24.625 0.0283617 0.0358042 2022-04-09T11:45:58.915 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 165 16 73791 73775 27.9415 30.875 0.0336846 0.0357814 2022-04-09T11:45:59.915 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 166 16 74308 74292 27.9678 32.3125 0.0300607 0.0357467 2022-04-09T11:46:00.915 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 167 16 74791 74775 27.9811 30.1875 0.0416171 0.0357311 2022-04-09T11:46:01.915 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 168 16 75279 75263 27.9961 30.5 0.0286021 0.0357092 2022-04-09T11:46:02.348 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:02.344074 I | op-osd: OSD orchestration status for PVC smithi119-lv-1-data-0vcsdm is "completed" 2022-04-09T11:46:02.349 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:02.344109 I | op-osd: creating OSD 0 on PVC "smithi119-lv-1-data-0vcsdm" 2022-04-09T11:46:02.349 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:02.344120 I | op-osd: OSD will have its main bluestore block on "smithi119-lv-1-data-0vcsdm" 2022-04-09T11:46:02.915 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 169 16 75701 75685 27.9865 26.375 0.0268452 0.0357129 2022-04-09T11:46:03.915 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 170 16 75912 75896 27.8994 13.1875 0.0178599 0.0357514 2022-04-09T11:46:04.915 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 171 16 76091 76075 27.8017 11.1875 0.123578 0.0359569 2022-04-09T11:46:05.569 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:05.565515 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-04-09T11:46:05.570 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:05.565531 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-04-09T11:46:05.723 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:05.717263 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-04-09T11:46:05.915 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 172 16 76469 76453 27.7774 23.625 0.0628653 0.0359925 2022-04-09T11:46:06.615 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:06.610781 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:46:06.915 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 173 16 76771 76755 27.7259 18.875 0.0065494 0.0359674 2022-04-09T11:46:06.957 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:06.953554 I | ceph-cluster-controller: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7... 2022-04-09T11:46:07.915 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 174 16 76771 76755 27.5666 0 - 0.0359674 2022-04-09T11:46:08.915 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 175 16 76771 76755 27.4091 0 - 0.0359674 2022-04-09T11:46:09.915 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 176 16 76771 76755 27.2533 0 - 0.0359674 2022-04-09T11:46:10.915 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 177 16 76771 76755 27.0994 0 - 0.0359674 2022-04-09T11:46:11.915 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 178 16 76771 76755 26.9471 0 - 0.0359674 2022-04-09T11:46:12.915 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 179 16 76771 76755 26.7966 0 - 0.0359674 2022-04-09T11:46:13.915 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:2022-04-09T11:46:13.912462+0000 min lat: 0.00166103 max lat: 9.73842 avg lat: 0.0359674 2022-04-09T11:46:13.916 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-04-09T11:46:13.916 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 180 16 76771 76755 26.6477 0 - 0.0359674 2022-04-09T11:46:14.916 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 181 16 76771 76755 26.5005 0 - 0.0359674 2022-04-09T11:46:15.916 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 182 16 76771 76755 26.3549 0 - 0.0359674 2022-04-09T11:46:16.916 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 183 16 76771 76755 26.2109 0 - 0.0359674 2022-04-09T11:46:17.916 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 184 16 76771 76755 26.0684 0 - 0.0359674 2022-04-09T11:46:18.916 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 185 16 76771 76755 25.9275 0 - 0.0359674 2022-04-09T11:46:19.916 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 186 16 76771 76755 25.7881 0 - 0.0359674 2022-04-09T11:46:20.916 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 187 16 76942 76926 25.7074 0.763393 0.0330321 0.0388906 2022-04-09T11:46:21.916 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 188 16 77346 77330 25.7049 25.25 0.0456778 0.038893 2022-04-09T11:46:22.448 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:22.444409 I | ceph-cluster-controller: detected ceph image version: "17.0.0-11488 quincy" 2022-04-09T11:46:22.448 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:22.444430 I | ceph-cluster-controller: validating ceph version from provided image 2022-04-09T11:46:22.449 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:22.444439 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11488 quincy", pursuing 2022-04-09T11:46:22.916 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 189 16 77738 77722 25.6985 24.5 0.0556864 0.0389039 2022-04-09T11:46:23.651 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:23.647349 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:46:23.658 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:23.651560 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-09T11:46:23.658 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:23.651742 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-09T11:46:23.916 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 190 16 78122 78106 25.6896 24 0.0542194 0.0389152 2022-04-09T11:46:24.412 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:24.404386 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11488 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7" 2022-04-09T11:46:24.510 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:24.506342 I | op-mon: start running mons 2022-04-09T11:46:24.530 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:24.526376 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:46:24.548 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:24.545337 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.24.44:6789"]}] data:a=10.107.24.44:6789 mapping:{"node":{"a":{"Name":"smithi119","Hostname":"smithi119","Address":"172.21.15.119"}}} maxMonId:0] 2022-04-09T11:46:24.557 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:24.554167 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-09T11:46:24.557 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:24.554371 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-09T11:46:24.916 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 191 16 78550 78534 25.6951 26.75 0.0349099 0.0389063 2022-04-09T11:46:25.141 INFO:teuthology.orchestra.run.smithi119.stderr:+ '[' 3 '!=' 4 ']' 2022-04-09T11:46:25.142 INFO:teuthology.orchestra.run.smithi119.stderr:+ echo 'waiting for osd to come back up' 2022-04-09T11:46:25.142 INFO:teuthology.orchestra.run.smithi119.stdout:waiting for osd to come back up 2022-04-09T11:46:25.143 INFO:teuthology.orchestra.run.smithi119.stderr:++ toolbox ceph osd stat 2022-04-09T11:46:25.143 INFO:teuthology.orchestra.run.smithi119.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-04-09T11:46:25.144 INFO:teuthology.orchestra.run.smithi119.stderr:++ cut -f3 -d ' ' 2022-04-09T11:46:25.519 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:25.516416 I | op-mon: targeting the mon count 1 2022-04-09T11:46:25.520 INFO:teuthology.orchestra.run.smithi119.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-04-09T11:46:25.597 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:25.591512 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-04-09T11:46:25.916 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 192 16 78977 78961 25.7003 26.6875 0.0585693 0.0389 2022-04-09T11:46:26.688 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:26.684190 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-04-09T11:46:26.689 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:26.684218 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-04-09T11:46:26.723 INFO:teuthology.orchestra.run.smithi119.stderr:+ num_osd=3 2022-04-09T11:46:26.724 INFO:teuthology.orchestra.run.smithi119.stderr:+ sleep 30s 2022-04-09T11:46:26.916 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 193 16 79320 79304 25.6782 21.4375 0.134023 0.0389211 2022-04-09T11:46:27.710 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:27.647511 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-04-09T11:46:27.711 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:27.647540 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-04-09T11:46:27.916 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 194 16 79767 79751 25.6898 27.9375 0.0420407 0.0389174 2022-04-09T11:46:28.548 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:28.510643 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-04-09T11:46:28.553 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:28.510661 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-04-09T11:46:28.916 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 195 16 80158 80142 25.6834 24.4375 0.11265 0.0389259 2022-04-09T11:46:29.152 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:29.148887 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-04-09T11:46:29.152 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:29.148912 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database 2022-04-09T11:46:29.921 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 196 16 80592 80576 25.6907 27.125 0.0314771 0.0389144 2022-04-09T11:46:29.928 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:29.921559 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database 2022-04-09T11:46:29.928 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:29.921583 I | op-config: deleting "log file" option from the mon configuration database 2022-04-09T11:46:30.546 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:30.542127 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-04-09T11:46:30.547 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:30.542153 I | op-mon: checking for basic quorum with existing mons 2022-04-09T11:46:30.653 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:30.646950 I | op-mon: mon "a" endpoint is [v2:10.107.24.44:3300,v1:10.107.24.44:6789] 2022-04-09T11:46:30.688 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:30.684112 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.107.24.44:6789"]}] data:a=10.107.24.44:6789 mapping:{"node":{"a":{"Name":"smithi119","Hostname":"smithi119","Address":"172.21.15.119"}}} maxMonId:0] 2022-04-09T11:46:30.711 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:30.708156 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-04-09T11:46:30.711 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:30.708340 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-04-09T11:46:30.918 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 197 16 81023 81007 25.6968 26.9375 0.0205113 0.0389063 2022-04-09T11:46:30.966 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:30.963090 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-04-09T11:46:30.983 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:30.980421 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-04-09T11:46:30.983 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:30.980449 I | op-mon: waiting for mon quorum with [a] 2022-04-09T11:46:31.471 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:31.467837 I | op-mon: mons running: [a] 2022-04-09T11:46:31.918 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 198 16 81423 81407 25.6933 25 0.0227784 0.0389126 2022-04-09T11:46:32.743 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:32.739669 I | op-mon: Monitors in quorum: [a] 2022-04-09T11:46:32.743 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:32.739695 I | op-mon: mons created: 1 2022-04-09T11:46:32.918 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 199 16 81786 81770 25.6782 22.6875 0.0305731 0.0389345 2022-04-09T11:46:33.573 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:33.570239 I | op-mon: waiting for mon quorum with [a] 2022-04-09T11:46:33.782 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:33.778115 I | op-mon: mons running: [a] 2022-04-09T11:46:33.921 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:2022-04-09T11:46:33.918600+0000 min lat: 0.00166103 max lat: 14.4109 avg lat: 0.0389612 2022-04-09T11:46:33.921 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-04-09T11:46:33.921 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 200 16 82143 82127 25.661 22.3125 0.0559391 0.0389612 2022-04-09T11:46:34.550 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:34.546729 I | op-mon: Monitors in quorum: [a] 2022-04-09T11:46:34.550 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:34.546812 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-04-09T11:46:34.921 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 201 16 82532 82516 25.6543 24.3125 0.0584523 0.0389696 2022-04-09T11:46:35.552 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:35.548790 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-04-09T11:46:35.921 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 202 16 82917 82901 25.6464 24.0625 0.0452997 0.0389785 2022-04-09T11:46:36.338 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:36.334664 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-04-09T11:46:36.921 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 203 16 83237 83221 25.6186 20 0.103686 0.039017 2022-04-09T11:46:37.035 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:37.032778 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-04-09T11:46:37.921 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 204 16 83631 83615 25.6137 24.625 0.016852 0.0390344 2022-04-09T11:46:38.385 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:38.379824 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-04-09T11:46:38.385 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:38.379849 I | cephclient: getting or creating ceph auth key "client.crash" 2022-04-09T11:46:38.921 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 205 16 84018 84002 25.6067 24.1875 0.0183695 0.0390443 2022-04-09T11:46:39.735 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:39.732237 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-04-09T11:46:39.921 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 206 16 84440 84424 25.6105 26.375 0.0416529 0.0390376 2022-04-09T11:46:40.570 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:40.566705 I | cephclient: successfully enabled msgr2 protocol 2022-04-09T11:46:40.570 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:40.566734 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-04-09T11:46:40.570 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:40.566739 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-04-09T11:46:40.921 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 207 16 84854 84838 25.6117 25.875 0.0654981 0.0390359 2022-04-09T11:46:41.400 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:41.393972 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-04-09T11:46:41.440 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:41.437227 I | op-mgr: start running mgr 2022-04-09T11:46:41.440 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:41.437254 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-04-09T11:46:41.921 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 208 16 85249 85233 25.6073 24.6875 0.0599498 0.0390402 2022-04-09T11:46:42.446 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:42.441223 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-04-09T11:46:42.455 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:42.453080 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-04-09T11:46:42.458 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:42.455260 I | op-mgr: setting services to point to mgr "a" 2022-04-09T11:46:42.574 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:42.569071 I | op-osd: start running osds in namespace "rook-ceph" 2022-04-09T11:46:42.575 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:42.569094 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-04-09T11:46:42.603 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:42.600487 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-04-09T11:46:42.607 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:42.603851 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi119-lv-2" 2022-04-09T11:46:42.607 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:42.603880 I | op-osd: OSD PVC "smithi119-lv-2-data-0d2cw5" already exists 2022-04-09T11:46:42.607 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:42.603889 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi119-lv-3" 2022-04-09T11:46:42.607 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:42.603896 I | op-osd: OSD PVC "smithi119-lv-3-data-05666c" already exists 2022-04-09T11:46:42.608 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:42.603903 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi119-lv-4" 2022-04-09T11:46:42.609 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:42.603910 I | op-osd: OSD PVC "smithi119-lv-4-data-0ts767" already exists 2022-04-09T11:46:42.609 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:42.603917 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi119-lv-1" 2022-04-09T11:46:42.610 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:42.603924 I | op-osd: OSD PVC "smithi119-lv-1-data-0vcsdm" already exists 2022-04-09T11:46:42.617 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:42.613908 I | op-osd: OSD will have its main bluestore block on "smithi119-lv-2-data-0d2cw5" 2022-04-09T11:46:42.617 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:42.613934 I | op-osd: OSD will have its main bluestore block on "smithi119-lv-3-data-05666c" 2022-04-09T11:46:42.618 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:42.613941 I | op-osd: OSD will have its main bluestore block on "smithi119-lv-4-data-0ts767" 2022-04-09T11:46:42.618 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:42.613945 I | op-osd: OSD will have its main bluestore block on "smithi119-lv-1-data-0vcsdm" 2022-04-09T11:46:42.618 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:42.613949 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-04-09T11:46:42.618 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:42.613953 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-04-09T11:46:42.921 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 209 16 85611 85595 25.593 22.625 0.0366562 0.0390582 2022-04-09T11:46:43.922 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 210 16 85906 85890 25.5589 18.4375 0.0372006 0.0391138 2022-04-09T11:46:44.063 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:44.056967 I | op-mgr: successful modules: dashboard 2022-04-09T11:46:44.092 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:44.083231 I | op-mgr: successful modules: mgr module(s) from the spec 2022-04-09T11:46:44.093 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:44.085076 I | op-mgr: successful modules: prometheus 2022-04-09T11:46:44.814 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:44.810693 I | op-mgr: successful modules: balancer 2022-04-09T11:46:44.869 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:44.866010 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-04-09T11:46:44.876 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:44.871627 I | op-osd: updating OSD 0 on PVC "smithi119-lv-1-data-0vcsdm" 2022-04-09T11:46:44.876 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:44.871650 I | op-osd: OSD will have its main bluestore block on "smithi119-lv-1-data-0vcsdm" 2022-04-09T11:46:44.922 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 211 16 86231 86215 25.534 20.3125 0.0295681 0.0391535 2022-04-09T11:46:45.922 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 212 16 86608 86592 25.5247 23.5625 0.0456457 0.0391636 2022-04-09T11:46:45.992 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:45.990092 I | op-mgr: successful modules: orchestrator modules 2022-04-09T11:46:46.701 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:46.698638 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-04-09T11:46:46.707 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:46.704490 I | op-osd: updating OSD 1 on PVC "smithi119-lv-4-data-0ts767" 2022-04-09T11:46:46.708 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:46.704520 I | op-osd: OSD will have its main bluestore block on "smithi119-lv-4-data-0ts767" 2022-04-09T11:46:46.922 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 213 16 86735 86719 25.4422 7.9375 0.0389154 0.0391734 2022-04-09T11:46:47.922 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 214 16 86735 86719 25.3233 0 - 0.0391734 2022-04-09T11:46:47.923 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:47.920429 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-04-09T11:46:47.928 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:47.925402 I | op-osd: updating OSD 2 on PVC "smithi119-lv-2-data-0d2cw5" 2022-04-09T11:46:47.928 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:47.925428 I | op-osd: OSD will have its main bluestore block on "smithi119-lv-2-data-0d2cw5" 2022-04-09T11:46:48.922 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 215 16 86735 86719 25.2055 0 - 0.0391734 2022-04-09T11:46:49.520 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:49.507489 W | cephclient: all OSDs are running on the same host. not performing upgrade check. running in best-effort 2022-04-09T11:46:49.521 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:49.513351 I | op-osd: updating OSD 3 on PVC "smithi119-lv-3-data-05666c" 2022-04-09T11:46:49.521 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:49.513372 I | op-osd: OSD will have its main bluestore block on "smithi119-lv-3-data-05666c" 2022-04-09T11:46:49.922 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 216 16 86735 86719 25.0888 0 - 0.0391734 2022-04-09T11:46:50.228 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:50.223525 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-04-09T11:46:50.232 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:46:50.223550 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-04-09T11:46:50.922 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 217 16 86843 86827 25.0043 1.6875 0.0395357 0.0399823 2022-04-09T11:46:51.922 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 218 16 87136 87120 24.9736 18.3125 0.0299601 0.0400258 2022-04-09T11:46:52.922 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 219 16 87428 87412 24.9429 18.25 0.080202 0.0400805 2022-04-09T11:46:53.923 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:2022-04-09T11:46:53.920561+0000 min lat: 0.00166103 max lat: 14.4109 avg lat: 0.0401579 2022-04-09T11:46:53.923 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-04-09T11:46:53.923 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 220 16 87662 87646 24.896 14.625 0.0708482 0.0401579 2022-04-09T11:46:54.923 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 221 16 87963 87947 24.8684 18.8125 0.0326502 0.040202 2022-04-09T11:46:55.923 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 222 16 88272 88256 24.8434 19.3125 0.0551067 0.0402368 2022-04-09T11:46:56.724 INFO:teuthology.orchestra.run.smithi119.stderr:+ '[' 3 '!=' 4 ']' 2022-04-09T11:46:56.725 INFO:teuthology.orchestra.run.smithi119.stdout:waiting for osd to come back up 2022-04-09T11:46:56.725 INFO:teuthology.orchestra.run.smithi119.stderr:+ echo 'waiting for osd to come back up' 2022-04-09T11:46:56.725 INFO:teuthology.orchestra.run.smithi119.stderr:++ toolbox ceph osd stat 2022-04-09T11:46:56.726 INFO:teuthology.orchestra.run.smithi119.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-04-09T11:46:56.726 INFO:teuthology.orchestra.run.smithi119.stderr:++ cut -f3 -d ' ' 2022-04-09T11:46:56.923 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 223 16 88463 88447 24.7855 11.9375 0.0492107 0.0402736 2022-04-09T11:46:57.038 INFO:teuthology.orchestra.run.smithi119.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-04-09T11:46:57.868 INFO:teuthology.orchestra.run.smithi119.stderr:+ num_osd=4 2022-04-09T11:46:57.868 INFO:teuthology.orchestra.run.smithi119.stderr:+ sleep 30s 2022-04-09T11:46:57.923 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 224 16 88732 88716 24.7499 16.8125 0.163391 0.04039 2022-04-09T11:46:58.923 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 225 16 89037 89021 24.7246 19.0625 0.0621317 0.0404341 2022-04-09T11:46:59.923 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 226 16 89241 89225 24.6717 12.75 0.0405458 0.0404744 2022-04-09T11:47:00.923 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 227 16 89467 89451 24.6252 14.125 0.0978268 0.0405993 2022-04-09T11:47:01.923 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 228 16 89690 89674 24.5783 13.9375 0.0655175 0.0406652 2022-04-09T11:47:02.923 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 229 16 89965 89949 24.546 17.1875 0.0353596 0.0407267 2022-04-09T11:47:03.924 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 230 16 90235 90219 24.5127 16.875 0.0465985 0.040785 2022-04-09T11:47:04.924 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 231 16 90522 90506 24.4842 17.9375 0.0737636 0.0408335 2022-04-09T11:47:05.924 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 232 16 90837 90821 24.4635 19.6875 0.0618733 0.0408661 2022-04-09T11:47:06.924 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 233 16 90967 90951 24.3934 8.125 0.00683385 0.0408937 2022-04-09T11:47:07.924 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 234 16 91262 91246 24.3679 18.4375 0.0459542 0.0410266 2022-04-09T11:47:08.924 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 235 16 91563 91547 24.3443 18.8125 0.0419993 0.0410652 2022-04-09T11:47:09.924 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 236 16 91862 91846 24.3203 18.6875 0.124802 0.0411077 2022-04-09T11:47:10.924 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 237 16 92166 92150 24.2978 19 0.0447336 0.041144 2022-04-09T11:47:11.924 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 238 16 92454 92438 24.2714 18 0.0475236 0.0411918 2022-04-09T11:47:12.924 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 239 16 92732 92716 24.2425 17.375 0.0574693 0.0412394 2022-04-09T11:47:13.924 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:2022-04-09T11:47:13.922593+0000 min lat: 0.00166103 max lat: 14.4109 avg lat: 0.0412942 2022-04-09T11:47:13.925 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-04-09T11:47:13.925 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 240 16 92996 92980 24.2103 16.5 0.0292131 0.0412942 2022-04-09T11:47:14.925 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 241 16 93123 93107 24.1427 7.9375 0.0158545 0.0413202 2022-04-09T11:47:15.925 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 242 16 93123 93107 24.043 0 - 0.0413202 2022-04-09T11:47:16.925 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 243 16 93334 93318 23.9983 6.59375 0.0970319 0.0416576 2022-04-09T11:47:17.925 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 244 16 93614 93598 23.9717 17.5 0.05258 0.0417023 2022-04-09T11:47:18.925 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 245 16 93844 93828 23.9325 14.375 0.0764089 0.0417709 2022-04-09T11:47:19.925 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 246 16 93845 93829 23.8355 0.0625 0.14034 0.041772 2022-04-09T11:47:20.925 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 247 16 94023 94007 23.784 11.125 0.0546893 0.042031 2022-04-09T11:47:21.925 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 248 16 94293 94277 23.7561 16.875 0.0235754 0.0420811 2022-04-09T11:47:22.925 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 249 16 94432 94416 23.6956 8.6875 0.0301055 0.0421885 2022-04-09T11:47:23.925 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 250 16 94653 94637 23.6561 13.8125 0.0610507 0.0422227 2022-04-09T11:47:24.926 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 251 16 94907 94891 23.6251 15.875 0.10789 0.0423177 2022-04-09T11:47:25.926 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 252 16 95184 95168 23.6 17.3125 0.13081 0.0423624 2022-04-09T11:47:26.926 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 253 16 95455 95439 23.5737 16.9375 0.0368822 0.0424065 2022-04-09T11:47:27.868 INFO:teuthology.orchestra.run.smithi119.stderr:+ '[' 4 '!=' 4 ']' 2022-04-09T11:47:27.870 INFO:teuthology.run_tasks:Running task rook.shell... 2022-04-09T11:47:27.885 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch status 2022-04-09T11:47:27.928 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 254 16 95741 95725 23.551 17.875 0.0730076 0.0424497 2022-04-09T11:47:28.928 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 255 16 96011 95995 23.5248 16.875 0.0934101 0.0424922 2022-04-09T11:47:29.928 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 256 16 96167 96151 23.471 9.75 0.0390234 0.0425938 2022-04-09T11:47:30.929 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 257 16 96465 96449 23.4521 18.625 0.0689766 0.0426287 2022-04-09T11:47:31.929 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 258 16 96715 96699 23.4218 15.625 0.14491 0.0426817 2022-04-09T11:47:32.231 INFO:teuthology.orchestra.run.smithi119.stdout:Backend: rook 2022-04-09T11:47:32.231 INFO:teuthology.orchestra.run.smithi119.stdout:Available: Yes 2022-04-09T11:47:32.279 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ps 2022-04-09T11:47:32.929 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 259 16 96961 96945 23.3907 15.375 0.0317967 0.0427406 2022-04-09T11:47:33.372 INFO:teuthology.orchestra.run.smithi119.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-04-09T11:47:33.372 INFO:teuthology.orchestra.run.smithi119.stdout:crashcollector.crash smithi119 running (8m) 0s ago 8m - - 2abf022dd4de 2022-04-09T11:47:33.373 INFO:teuthology.orchestra.run.smithi119.stdout:mgr.a smithi119 running (9m) 0s ago 9m - - 2abf022dd4de 2022-04-09T11:47:33.373 INFO:teuthology.orchestra.run.smithi119.stdout:mon.a smithi119 running (10m) 0s ago 10m - - 2abf022dd4de 2022-04-09T11:47:33.373 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0 smithi119 running (89s) 0s ago 90s - - 2abf022dd4de 2022-04-09T11:47:33.373 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1 smithi119 running (8m) 0s ago 8m - - 2abf022dd4de 2022-04-09T11:47:33.374 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2 smithi119 running (8m) 0s ago 8m - - 2abf022dd4de 2022-04-09T11:47:33.374 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3 smithi119 running (7m) 0s ago 7m - - 2abf022dd4de 2022-04-09T11:47:33.415 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls 2022-04-09T11:47:33.929 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:2022-04-09T11:47:33.927209+0000 min lat: 0.00166103 max lat: 14.4109 avg lat: 0.0427964 2022-04-09T11:47:33.929 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-04-09T11:47:33.930 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 260 15 97208 97193 23.3604 15.5 0.0926213 0.0427964 2022-04-09T11:47:34.289 INFO:teuthology.orchestra.run.smithi119.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-04-09T11:47:34.289 INFO:teuthology.orchestra.run.smithi119.stdout:crash 1/1 0s ago 8m * 2022-04-09T11:47:34.289 INFO:teuthology.orchestra.run.smithi119.stdout:mgr 1/1 0s ago 9m count:1 2022-04-09T11:47:34.289 INFO:teuthology.orchestra.run.smithi119.stdout:mon 1/1 0s ago 10m count:1 2022-04-09T11:47:34.290 INFO:teuthology.orchestra.run.smithi119.stdout:osd 8 0s ago 8m 2022-04-09T11:47:34.290 INFO:teuthology.orchestra.run.smithi119.stdout:osd.all-available-devices 0 0s ago - * 2022-04-09T11:47:34.347 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch host ls 2022-04-09T11:47:34.929 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 261 16 97451 97435 23.3288 15.125 0.0432875 0.0428545 2022-04-09T11:47:35.217 INFO:teuthology.orchestra.run.smithi119.stdout:HOST ADDR LABELS STATUS 2022-04-09T11:47:35.218 INFO:teuthology.orchestra.run.smithi119.stdout:smithi119 172.21.15.119/smithi119 2022-04-09T11:47:35.218 INFO:teuthology.orchestra.run.smithi119.stdout:1 hosts in cluster 2022-04-09T11:47:35.246 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch device ls 2022-04-09T11:47:35.929 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 262 16 97703 97687 23.2999 15.75 0.0332895 0.042906 2022-04-09T11:47:36.124 INFO:teuthology.orchestra.run.smithi119.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-04-09T11:47:36.125 INFO:teuthology.orchestra.run.smithi119.stdout:smithi119 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-04-09T11:47:36.125 INFO:teuthology.orchestra.run.smithi119.stdout:smithi119 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-04-09T11:47:36.125 INFO:teuthology.orchestra.run.smithi119.stdout:smithi119 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-04-09T11:47:36.126 INFO:teuthology.orchestra.run.smithi119.stdout:smithi119 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-04-09T11:47:36.171 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch apply rgw foo 2022-04-09T11:47:36.929 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 263 16 97927 97911 23.2645 14 0.0570179 0.0429551 2022-04-09T11:47:36.944 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:47:36.942259 I | ceph-spec: adding finalizer "cephobjectstore.ceph.rook.io" on "foo" 2022-04-09T11:47:36.946 INFO:teuthology.orchestra.run.smithi119.stdout:Created 2022-04-09T11:47:36.963 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch apply mds foo 2022-04-09T11:47:37.479 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:47:37.475860 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:47:37.929 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 264 16 98076 98060 23.2117 9.3125 0.0622997 0.0430715 2022-04-09T11:47:38.382 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:47:38.380319 I | ceph-spec: adding finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-04-09T11:47:38.384 INFO:teuthology.orchestra.run.smithi119.stdout:Created 2022-04-09T11:47:38.419 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:47:38.416495 W | ceph-file-controller: failed to set filesystem "foo" status to "". failed to update object "rook-ceph/foo" status: Operation cannot be fulfilled on cephfilesystems.ceph.rook.io "foo": the object has been modified; please apply your changes to the latest version and try again 2022-04-09T11:47:38.437 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch apply rbd-mirror 2022-04-09T11:47:38.451 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:47:38.446241 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:47:38.929 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 265 16 98293 98277 23.1752 13.5625 0.0357066 0.0431331 2022-04-09T11:47:39.444 INFO:teuthology.orchestra.run.smithi119.stdout:Success 2022-04-09T11:47:39.477 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:47:39.471656 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:47:39.497 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch apply nfs foo --port 12777 2022-04-09T11:47:39.929 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 266 16 98483 98467 23.1328 11.875 0.0288668 0.0432155 2022-04-09T11:47:40.049 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:47:40.033316 I | ceph-object-controller: reconciling object store deployments 2022-04-09T11:47:40.078 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:47:40.073909 I | ceph-object-controller: ceph object store gateway service running at 10.108.92.150 2022-04-09T11:47:40.079 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:47:40.073932 I | ceph-object-controller: reconciling object store pools 2022-04-09T11:47:40.384 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:47:40.378971 W | ceph-rbd-mirror-controller: (DEPRECATED) use of peer secret names in CephRBDMirror is deprecated. Please use CephBlockPool CR to configure peer secret names and import peers. 2022-04-09T11:47:40.385 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:47:40.379002 I | ceph-rbd-mirror-controller: configure rbd-mirroring with 1 workers 2022-04-09T11:47:40.385 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:47:40.379014 I | cephclient: getting or creating ceph auth key "client.rbd-mirror.a" 2022-04-09T11:47:40.930 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 267 16 98655 98639 23.0864 10.75 0.307431 0.0433023 2022-04-09T11:47:41.257 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:47:41.253938 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-04-09T11:47:41.930 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 268 16 98859 98843 23.0478 12.75 0.1466 0.0433735 2022-04-09T11:47:42.497 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:47:42.484821 I | ceph-rbd-mirror-controller: "rook-ceph-rbd-mirror-a" deployment started 2022-04-09T11:47:42.837 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:47:42.831524 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi119": the object has been modified; please apply your changes to the latest version and try again 2022-04-09T11:47:42.930 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 269 16 98987 98971 22.9919 8 0.0909336 0.0434784 2022-04-09T11:47:43.230 INFO:teuthology.orchestra.run.smithi119.stdout:Unable to create NFS daemon, check logs for more traceback 2022-04-09T11:47:43.231 INFO:teuthology.orchestra.run.smithi119.stdout:'RookOrchestrator' object has no attribute 'open_ioctx' 2022-04-09T11:47:43.259 DEBUG:teuthology.run_tasks:Unwinding manager radosbench 2022-04-09T11:47:43.272 INFO:tasks.radosbench:joining radosbench (timing out after 11100s) 2022-04-09T11:47:43.272 INFO:teuthology.orchestra.run:waiting for 11100 2022-04-09T11:47:43.930 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 270 16 99126 99110 22.9389 8.6875 0.0633554 0.0435812 2022-04-09T11:47:44.413 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:47:44.410396 I | ceph-file-controller: creating filesystem "foo" 2022-04-09T11:47:44.930 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 271 16 99211 99195 22.8738 5.3125 0.018911 0.043625 2022-04-09T11:47:45.930 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 272 16 99228 99212 22.7937 1.0625 0.0223436 0.0437451 2022-04-09T11:47:46.931 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 273 16 99228 99212 22.7101 0 - 0.0437451 2022-04-09T11:47:47.931 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 274 16 99463 99447 22.6808 7.34375 0.0458349 0.0440787 2022-04-09T11:47:48.931 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 275 16 99586 99570 22.6263 7.6875 0.0236455 0.0441067 2022-04-09T11:47:49.931 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 276 16 99595 99579 22.5464 0.5625 0.0278436 0.0442249 2022-04-09T11:47:50.931 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 277 16 99816 99800 22.5148 13.8125 0.0553048 0.0443898 2022-04-09T11:47:51.931 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 278 16 99957 99941 22.4656 8.8125 0.00876556 0.0444493 2022-04-09T11:47:52.932 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 279 16 99978 99962 22.3897 1.3125 0.0301992 0.0446067 2022-04-09T11:47:53.932 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:2022-04-09T11:47:53.929707+0000 min lat: 0.00166103 max lat: 14.4109 avg lat: 0.0447232 2022-04-09T11:47:53.932 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-04-09T11:47:53.932 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 280 16 100179 100163 22.3546 12.5625 0.0427464 0.0447232 2022-04-09T11:47:54.932 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 281 16 100435 100419 22.332 16 0.0678589 0.0447669 2022-04-09T11:47:55.932 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 282 16 100666 100650 22.304 14.4375 0.148378 0.0448208 2022-04-09T11:47:56.932 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 283 16 100898 100882 22.2764 14.5 0.062895 0.0448798 2022-04-09T11:47:57.142 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:47:57.139865 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.buckets.non-ec" 2022-04-09T11:47:57.172 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:47:57.167607 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.meta" 2022-04-09T11:47:57.932 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 284 16 101095 101079 22.2414 12.3125 0.0996805 0.0449515 2022-04-09T11:47:58.145 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:47:58.141671 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.control" 2022-04-09T11:47:58.163 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:47:58.157424 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.buckets.index" 2022-04-09T11:47:58.207 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:47:58.197850 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo.rgw.log" 2022-04-09T11:47:58.932 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 285 16 101291 101275 22.2063 12.25 0.0845043 0.0450219 2022-04-09T11:47:59.932 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 286 16 101542 101526 22.1835 15.6875 0.0266467 0.0450683 2022-04-09T11:48:00.932 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 287 16 101784 101768 22.1589 15.125 0.0468941 0.0451164 2022-04-09T11:48:01.157 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:01.154271 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo-metadata" 2022-04-09T11:48:01.933 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 288 16 101972 101956 22.1228 11.75 0.0486431 0.0451891 2022-04-09T11:48:02.933 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 289 16 102116 102100 22.0773 9 0.0676827 0.0452817 2022-04-09T11:48:03.933 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 290 16 102358 102342 22.0534 15.125 0.121058 0.0453342 2022-04-09T11:48:04.201 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:04.198696 I | cephclient: creating replicated pool foo.rgw.buckets.index succeeded 2022-04-09T11:48:04.215 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:04.210919 I | cephclient: creating replicated pool foo.rgw.control succeeded 2022-04-09T11:48:04.228 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:04.212989 I | cephclient: creating replicated pool foo.rgw.log succeeded 2022-04-09T11:48:04.235 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:04.230686 I | cephclient: creating replicated pool foo.rgw.buckets.non-ec succeeded 2022-04-09T11:48:04.235 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:04.230784 I | cephclient: creating replicated pool foo.rgw.meta succeeded 2022-04-09T11:48:04.280 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:04.277173 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-04-09T11:48:04.281 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:04.277199 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-04-09T11:48:04.291 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:04.285840 I | cephclient: creating replicated pool foo-metadata succeeded 2022-04-09T11:48:04.298 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:04.293218 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:48:04.933 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 291 16 102556 102540 22.0201 12.375 0.0282728 0.0454014 2022-04-09T11:48:05.933 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 292 16 102753 102737 21.9869 12.3125 0.117599 0.0454667 2022-04-09T11:48:06.933 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 293 16 102933 102917 21.9502 11.25 0.140024 0.0455457 2022-04-09T11:48:07.933 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 294 16 103156 103140 21.923 13.9375 0.0364273 0.0456032 2022-04-09T11:48:08.353 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:08.343912 I | ceph-object-controller: reconciling object store deployments 2022-04-09T11:48:08.933 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 295 16 103366 103350 21.8931 13.125 0.0306591 0.0456607 2022-04-09T11:48:09.933 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 296 15 103598 103583 21.8684 14.5625 0.122551 0.0457164 2022-04-09T11:48:09.968 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:09.965958 I | ceph-object-controller: ceph object store gateway service running at 10.108.92.150 2022-04-09T11:48:09.969 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:09.965985 I | ceph-object-controller: reconciling object store pools 2022-04-09T11:48:10.827 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:10.824122 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-04-09T11:48:10.933 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 297 16 103827 103811 21.8427 14.25 0.0342572 0.0457689 2022-04-09T11:48:11.934 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 298 16 103979 103963 21.8013 9.5 0.0598973 0.0458591 2022-04-09T11:48:12.934 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 299 16 104202 104186 21.775 13.9375 0.0599295 0.0459087 2022-04-09T11:48:13.420 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:13.417539 I | cephclient: setting pool property "compression_mode" to "none" on pool "foo-data0" 2022-04-09T11:48:13.934 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:2022-04-09T11:48:13.931711+0000 min lat: 0.00166103 max lat: 14.4109 avg lat: 0.04597 2022-04-09T11:48:13.934 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-04-09T11:48:13.934 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 300 16 104383 104367 21.7401 11.3125 0.262979 0.04597 2022-04-09T11:48:14.886 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:14.822102 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-04-09T11:48:14.886 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:14.838043 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:48:14.934 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 301 16 104625 104609 21.7181 15.125 0.027995 0.0460348 2022-04-09T11:48:15.226 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:15.223459 I | cephclient: creating replicated pool foo-data0 succeeded 2022-04-09T11:48:15.227 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:15.223492 I | cephclient: creating filesystem "foo" with metadata pool "foo-metadata" and data pools [foo-data0] 2022-04-09T11:48:15.934 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 302 16 104848 104832 21.6924 13.9375 0.146379 0.0460839 2022-04-09T11:48:16.934 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 303 16 105058 105042 21.6641 13.125 0.0603687 0.046151 2022-04-09T11:48:17.934 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 304 16 105294 105278 21.6413 14.75 0.0547553 0.0461983 2022-04-09T11:48:18.599 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:18.593395 I | ceph-object-controller: reconciling object store deployments 2022-04-09T11:48:18.814 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:18.759758 I | ceph-file-controller: created filesystem "foo" on 1 data pool(s) and metadata pool "foo-metadata" 2022-04-09T11:48:18.934 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 305 16 105483 105467 21.6091 11.8125 0.112029 0.0462627 2022-04-09T11:48:19.465 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:19.460478 I | cephclient: setting allow_standby_replay for filesystem "foo" 2022-04-09T11:48:19.743 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:19.732842 I | ceph-object-controller: ceph object store gateway service running at 10.108.92.150 2022-04-09T11:48:19.743 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:19.732862 I | ceph-object-controller: reconciling object store pools 2022-04-09T11:48:19.934 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 306 16 105728 105712 21.5885 15.3125 0.0877641 0.0463107 2022-04-09T11:48:20.377 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:20.372922 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-04-09T11:48:20.935 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 307 16 105915 105899 21.5563 11.6875 0.130735 0.0463687 2022-04-09T11:48:21.151 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:21.147793 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi119": the object has been modified; please apply your changes to the latest version and try again 2022-04-09T11:48:21.249 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:21.245631 I | ceph-file-controller: start running mdses for filesystem "foo" 2022-04-09T11:48:21.935 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 308 16 106073 106057 21.5183 9.875 0.0177499 0.0464574 2022-04-09T11:48:22.935 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 309 16 106290 106274 21.4926 13.5625 0.0429598 0.0465133 2022-04-09T11:48:22.956 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:22.949959 I | cephclient: getting or creating ceph auth key "mds.foo-a" 2022-04-09T11:48:23.935 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 310 16 106498 106482 21.4652 13 0.0428162 0.0465757 2022-04-09T11:48:24.935 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 311 16 106747 106731 21.4462 15.5625 0.0747572 0.0466158 2022-04-09T11:48:25.182 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:25.176058 I | op-mds: setting mds config flags 2022-04-09T11:48:25.182 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:25.176088 I | op-config: setting "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-04-09T11:48:25.194 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:25.191950 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-04-09T11:48:25.241 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:25.238073 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:48:25.900 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:25.897277 I | op-config: successfully set "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-04-09T11:48:25.935 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 312 16 106974 106958 21.4229 14.1875 0.0756432 0.0466678 2022-04-09T11:48:25.945 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:25.941655 I | cephclient: getting or creating ceph auth key "mds.foo-b" 2022-04-09T11:48:26.935 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 313 16 107144 107128 21.3884 10.625 0.0301113 0.0467412 2022-04-09T11:48:27.567 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:27.563902 I | op-mds: setting mds config flags 2022-04-09T11:48:27.567 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:27.563929 I | op-config: setting "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-04-09T11:48:27.760 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:27.738859 I | ceph-object-controller: reconciling object store deployments 2022-04-09T11:48:27.935 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 314 16 107356 107340 21.3625 13.25 0.0792627 0.0467999 2022-04-09T11:48:28.935 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 315 16 107588 107572 21.3407 14.5 0.0183988 0.0468477 2022-04-09T11:48:29.266 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:29.263262 I | ceph-object-controller: ceph object store gateway service running at 10.108.92.150 2022-04-09T11:48:29.266 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:29.263281 I | ceph-object-controller: reconciling object store pools 2022-04-09T11:48:29.936 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 316 16 107808 107792 21.3167 13.75 0.102184 0.0469016 2022-04-09T11:48:30.583 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:30.574829 I | op-config: successfully set "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-04-09T11:48:30.717 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:30.708984 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-04-09T11:48:30.936 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 317 16 108035 108019 21.2942 14.1875 0.0818782 0.0469405 2022-04-09T11:48:31.936 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 318 16 108194 108178 21.2585 9.9375 0.0815004 0.0470286 2022-04-09T11:48:32.936 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 319 16 108365 108349 21.2253 10.6875 0.0823293 0.0470847 2022-04-09T11:48:33.936 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:2022-04-09T11:48:33.933883+0000 min lat: 0.00166103 max lat: 14.4109 avg lat: 0.0471551 2022-04-09T11:48:33.936 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-04-09T11:48:33.937 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 320 16 108582 108566 21.2014 13.5625 0.0244839 0.0471551 2022-04-09T11:48:34.936 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 321 16 108835 108819 21.1846 15.8125 0.0282613 0.0471928 2022-04-09T11:48:35.936 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 322 16 109082 109066 21.1667 15.4375 0.0604614 0.0472342 2022-04-09T11:48:36.936 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 323 16 109231 109215 21.13 9.3125 0.201524 0.0472942 2022-04-09T11:48:37.070 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:37.067476 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-04-09T11:48:37.120 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:37.116199 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:48:37.937 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 324 16 109393 109377 21.0961 10.125 0.055916 0.0473925 2022-04-09T11:48:38.937 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 325 16 109587 109571 21.0685 12.125 0.145715 0.0474531 2022-04-09T11:48:39.937 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 326 16 109867 109851 21.0575 17.5 0.0517469 0.0474783 2022-04-09T11:48:40.383 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:40.370850 I | ceph-object-controller: reconciling object store deployments 2022-04-09T11:48:40.937 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 327 16 110173 110157 21.0516 19.125 0.0142768 0.0474912 2022-04-09T11:48:41.870 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:41.865295 I | ceph-object-controller: ceph object store gateway service running at 10.108.92.150 2022-04-09T11:48:41.871 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:41.865383 I | ceph-object-controller: reconciling object store pools 2022-04-09T11:48:41.937 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 328 16 110532 110516 21.0558 22.4375 0.030187 0.0474832 2022-04-09T11:48:42.937 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 329 16 110754 110738 21.034 13.875 0.0501344 0.0475328 2022-04-09T11:48:43.937 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 330 16 110994 110978 21.0157 15 0.0226162 0.0475613 2022-04-09T11:48:44.753 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:44.748134 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-04-09T11:48:44.938 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 331 16 111308 111292 21.0115 19.625 0.0537024 0.0475785 2022-04-09T11:48:45.938 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 332 16 111489 111473 20.9823 11.3125 0.0392417 0.0476505 2022-04-09T11:48:46.938 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 333 16 111745 111729 20.9673 16 0.0175645 0.0476835 2022-04-09T11:48:47.938 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 334 16 112030 112014 20.9578 17.8125 0.100229 0.0477037 2022-04-09T11:48:48.938 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 335 16 112387 112371 20.9619 22.3125 0.127527 0.0476949 2022-04-09T11:48:49.938 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 336 16 112589 112573 20.9371 12.625 0.0536781 0.0477544 2022-04-09T11:48:50.938 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 337 16 112939 112923 20.9398 21.875 0.0262395 0.0477477 2022-04-09T11:48:51.695 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:51.692263 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-04-09T11:48:51.801 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:51.783903 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:48:51.938 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 338 16 113198 113182 20.9258 16.1875 0.0173657 0.0477755 2022-04-09T11:48:52.938 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 339 16 113539 113523 20.9269 21.3125 0.0295074 0.0477768 2022-04-09T11:48:53.791 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:53.788150 I | ceph-object-controller: reconciling object store deployments 2022-04-09T11:48:53.939 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:2022-04-09T11:48:53.936250+0000 min lat: 0.00166103 max lat: 14.4109 avg lat: 0.0477822 2022-04-09T11:48:53.939 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2022-04-09T11:48:53.939 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 340 16 113860 113844 20.9244 20.0625 0.0663088 0.0477822 2022-04-09T11:48:54.939 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 341 16 114190 114174 20.9235 20.625 0.0173097 0.0477831 2022-04-09T11:48:55.939 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 342 16 114535 114519 20.9253 21.5625 0.0522603 0.0477798 2022-04-09T11:48:56.154 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:56.150059 I | ceph-object-controller: ceph object store gateway service running at 10.108.92.150 2022-04-09T11:48:56.154 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:56.150083 I | ceph-object-controller: reconciling object store pools 2022-04-09T11:48:56.801 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:56.797890 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-04-09T11:48:56.939 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 343 16 114852 114836 20.9221 19.8125 0.142689 0.0477825 2022-04-09T11:48:57.939 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 344 16 115070 115054 20.9009 13.625 0.167335 0.0478295 2022-04-09T11:48:58.939 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 345 16 115368 115352 20.8943 18.625 0.0097475 0.0478502 2022-04-09T11:48:59.498 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:59.494362 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-04-09T11:48:59.672 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:48:59.666167 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:48:59.939 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 346 16 115712 115696 20.896 21.5 0.0490938 0.0478479 2022-04-09T11:49:00.939 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 347 16 116059 116043 20.8983 21.6875 0.0373166 0.0478418 2022-04-09T11:49:01.697 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:49:01.693014 I | ceph-object-controller: reconciling object store deployments 2022-04-09T11:49:01.771 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:49:01.768143 I | ceph-object-controller: ceph object store gateway service running at 10.108.92.150 2022-04-09T11:49:01.771 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:49:01.768163 I | ceph-object-controller: reconciling object store pools 2022-04-09T11:49:01.939 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 348 16 116408 116392 20.9009 21.8125 0.0414428 0.0478362 2022-04-09T11:49:02.307 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:49:02.302362 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-04-09T11:49:02.940 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 349 16 116654 116638 20.8851 15.375 0.126008 0.0478607 2022-04-09T11:49:03.940 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 350 16 116933 116917 20.8752 17.4375 0.0241209 0.0478948 2022-04-09T11:49:04.940 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 351 16 117267 117251 20.8752 20.875 0.022671 0.0478948 2022-04-09T11:49:05.272 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:49:05.269061 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-04-09T11:49:05.602 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:49:05.598402 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:49:05.940 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 352 16 117638 117622 20.8818 23.1875 0.0162811 0.0478779 2022-04-09T11:49:06.940 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 353 16 117767 117751 20.8455 8.0625 0.0448365 0.0479621 2022-04-09T11:49:07.464 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:49:07.459669 I | ceph-object-controller: reconciling object store deployments 2022-04-09T11:49:07.553 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:49:07.548753 I | ceph-object-controller: ceph object store gateway service running at 10.108.92.150 2022-04-09T11:49:07.557 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:49:07.548774 I | ceph-object-controller: reconciling object store pools 2022-04-09T11:49:07.940 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 354 16 118134 118118 20.8514 22.9375 0.0448024 0.0479499 2022-04-09T11:49:08.198 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:49:08.190723 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-04-09T11:49:08.940 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 355 16 118409 118393 20.841 17.1875 0.0576864 0.0479706 2022-04-09T11:49:09.940 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 356 16 118729 118713 20.8387 20 0.0636387 0.0479783 2022-04-09T11:49:10.941 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 357 16 119090 119074 20.8435 22.5625 0.0321956 0.0479683 2022-04-09T11:49:11.386 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:49:11.382795 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-04-09T11:49:11.941 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 358 16 119436 119420 20.8457 21.625 0.0282932 0.047963 2022-04-09T11:49:12.042 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:49:12.035224 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:49:12.941 INFO:tasks.radosbench.radosbench.a.smithi119.stdout: 359 16 119773 119757 20.8463 21.0625 0.130598 0.047959 2022-04-09T11:49:13.941 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:Total time run: 360.025 2022-04-09T11:49:13.942 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:Total writes made: 120009 2022-04-09T11:49:13.942 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:Write size: 65536 2022-04-09T11:49:13.942 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:Object size: 65536 2022-04-09T11:49:13.942 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:Bandwidth (MB/sec): 20.8334 2022-04-09T11:49:13.943 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:Stddev Bandwidth: 15.3488 2022-04-09T11:49:13.943 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:Max bandwidth (MB/sec): 89.3125 2022-04-09T11:49:13.943 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:Min bandwidth (MB/sec): 0 2022-04-09T11:49:13.943 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:Average IOPS: 333 2022-04-09T11:49:13.943 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:Stddev IOPS: 245.591 2022-04-09T11:49:13.944 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:Max IOPS: 1429 2022-04-09T11:49:13.944 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:Min IOPS: 0 2022-04-09T11:49:13.944 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:Average Latency(s): 0.0479927 2022-04-09T11:49:13.944 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:Stddev Latency(s): 0.267399 2022-04-09T11:49:13.944 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:Max latency(s): 14.4109 2022-04-09T11:49:13.945 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:Min latency(s): 0.00166103 2022-04-09T11:49:13.945 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:Cleaning up (deleting benchmark objects) 2022-04-09T11:49:14.034 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:49:14.022929 I | ceph-object-controller: reconciling object store deployments 2022-04-09T11:49:14.121 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:49:14.118277 I | ceph-object-controller: ceph object store gateway service running at 10.108.92.150 2022-04-09T11:49:14.122 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:49:14.118302 I | ceph-object-controller: reconciling object store pools 2022-04-09T11:49:14.778 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:49:14.774700 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-04-09T11:49:17.566 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:49:17.560090 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-04-09T11:49:18.851 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:49:18.846688 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:49:21.022 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:49:21.017865 I | ceph-object-controller: reconciling object store deployments 2022-04-09T11:49:22.031 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:49:22.027006 I | ceph-object-controller: ceph object store gateway service running at 10.108.92.150 2022-04-09T11:49:22.031 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:49:22.027034 I | ceph-object-controller: reconciling object store pools 2022-04-09T11:49:22.653 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:49:22.648918 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-04-09T11:49:25.740 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:49:25.737082 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-04-09T11:49:28.311 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:49:28.306763 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:49:31.557 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:49:31.553586 I | ceph-object-controller: reconciling object store deployments 2022-04-09T11:49:34.301 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:49:34.297856 I | ceph-object-controller: ceph object store gateway service running at 10.108.92.150 2022-04-09T11:49:34.301 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:49:34.297875 I | ceph-object-controller: reconciling object store pools 2022-04-09T11:49:35.115 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:49:34.990774 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-04-09T11:49:37.568 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:49:37.558724 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-04-09T11:49:42.861 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:49:42.857552 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:49:44.894 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:49:44.890503 I | ceph-object-controller: reconciling object store deployments 2022-04-09T11:49:45.028 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:49:45.022915 I | ceph-object-controller: ceph object store gateway service running at 10.108.92.150 2022-04-09T11:49:45.028 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:49:45.022941 I | ceph-object-controller: reconciling object store pools 2022-04-09T11:49:45.606 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:49:45.600331 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-04-09T11:49:48.280 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:49:48.276808 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-04-09T11:49:58.775 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:49:58.769344 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:50:00.978 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:50:00.972724 I | ceph-object-controller: reconciling object store deployments 2022-04-09T11:50:01.260 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:50:01.255181 I | ceph-object-controller: ceph object store gateway service running at 10.108.92.150 2022-04-09T11:50:01.260 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:50:01.255204 I | ceph-object-controller: reconciling object store pools 2022-04-09T11:50:01.823 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:50:01.819134 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-04-09T11:50:04.657 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:50:04.653208 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-04-09T11:50:25.143 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:50:25.139742 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:50:27.721 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:50:27.716094 I | ceph-object-controller: reconciling object store deployments 2022-04-09T11:50:27.877 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:50:27.872557 I | ceph-object-controller: ceph object store gateway service running at 10.108.92.150 2022-04-09T11:50:27.877 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:50:27.872586 I | ceph-object-controller: reconciling object store pools 2022-04-09T11:50:28.454 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:50:28.449742 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-04-09T11:50:31.132 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:50:31.127016 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-04-09T11:51:12.103 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:51:12.096459 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:51:14.227 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:51:14.220914 I | ceph-object-controller: reconciling object store deployments 2022-04-09T11:51:14.420 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:51:14.414963 I | ceph-object-controller: ceph object store gateway service running at 10.108.92.150 2022-04-09T11:51:14.420 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:51:14.414992 I | ceph-object-controller: reconciling object store pools 2022-04-09T11:51:14.878 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:51:14.873549 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-04-09T11:51:17.703 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:51:17.695037 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-04-09T11:52:39.648 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:52:39.640910 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:52:41.714 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:52:41.699218 I | ceph-object-controller: reconciling object store deployments 2022-04-09T11:52:41.818 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:52:41.812840 I | ceph-object-controller: ceph object store gateway service running at 10.108.92.150 2022-04-09T11:52:41.818 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:52:41.812864 I | ceph-object-controller: reconciling object store pools 2022-04-09T11:52:42.367 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:52:42.362555 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-04-09T11:52:46.265 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:52:46.253042 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-04-09T11:55:30.115 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:55:30.103818 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:55:32.998 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:55:32.951724 I | ceph-object-controller: reconciling object store deployments 2022-04-09T11:55:34.036 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:55:34.029382 I | ceph-object-controller: ceph object store gateway service running at 10.108.92.150 2022-04-09T11:55:34.037 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:55:34.029407 I | ceph-object-controller: reconciling object store pools 2022-04-09T11:55:34.624 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:55:34.618880 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-04-09T11:55:37.293 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:55:37.288287 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool .rgw.root for object store foo.: failed to set size property to replicated pool ".rgw.root" to 3: failed to set pool property "size" on pool ".rgw.root": exit status 2 2022-04-09T11:55:41.177 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:Removed 120009 objects 2022-04-09T11:55:41.177 INFO:tasks.radosbench.radosbench.a.smithi119.stdout:Clean up completed and total clean up time :387.229 2022-04-09T11:55:45.270 INFO:tasks.rook.ceph_manager.ceph:removing pool_name unique_pool_0 2022-04-09T11:55:45.271 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2022-04-09T11:55:48.447 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'unique_pool_0' removed 2022-04-09T11:55:48.468 DEBUG:teuthology.run_tasks:Unwinding manager install 2022-04-09T11:55:48.479 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2022-04-09T11:55:48.480 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2022-04-09T11:55:49.039 DEBUG:teuthology.run_tasks:Unwinding manager rook 2022-04-09T11:55:49.049 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T11:55:50.585 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:55:50.586 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:47:42.000000Z", "last_refresh": "2022-04-09T11:55:50.360724Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:55:50.360724Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T11:55:50.360724Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T11:55:50.360724Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T11:55:50.360724Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:55:50.360724Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_name": "rbd-mirror", "service_type": "rbd-mirror", "status": {"last_refresh": "2022-04-09T11:55:50.360724Z", "running": 0, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:55:50.360724Z", "running": 0, "size": 1}}] 2022-04-09T11:55:50.610 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch rm mds.foo 2022-04-09T11:55:51.539 INFO:teuthology.orchestra.run.smithi119.stdout:Removed cephfilesystems/foo 2022-04-09T11:55:51.561 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch rm rbd-mirror 2022-04-09T11:55:51.601 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:55:51.595718 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:55:52.215 INFO:teuthology.orchestra.run.smithi119.stdout:Removed cephrbdmirrors/default-rbd-mirror 2022-04-09T11:55:52.234 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:55:52.228707 I | ceph-spec: object "rook-ceph-rbd-mirror-a" matched on delete, reconciling 2022-04-09T11:55:52.269 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch rm rgw.foo 2022-04-09T11:55:52.876 INFO:teuthology.orchestra.run.smithi119.stdout:Removed cephobjectstores/foo 2022-04-09T11:55:52.895 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T11:55:53.180 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:55:53.174801 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:55:53.762 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:55:53.756890 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-04-09T11:55:56.902 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:55:56.896138 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-04-09T11:55:56.902 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:55:56.896161 I | op-mds: successfully deleted mds config for "mds.foo-a" in mon configuration database 2022-04-09T11:55:56.902 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:55:56.896167 I | cephclient: deleting ceph auth "mds.foo-a" 2022-04-09T11:55:56.906 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:55:56.906 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:47:42.000000Z", "last_refresh": "2022-04-09T11:55:55.806821Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:55:55.806821Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T11:55:55.806821Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T11:55:55.806821Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T11:55:55.806821Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:55:55.806821Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:55:55.806821Z", "running": 0, "size": 1}}] 2022-04-09T11:55:57.461 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:55:57.455638 I | op-mds: successfully deleted mds CephX key for "mds.foo-a" 2022-04-09T11:55:58.539 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:55:58.533525 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-04-09T11:55:58.966 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:55:58.959995 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T11:55:58.943+0000 7f570b800340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T11:55:58.966 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T11:55:58.943+0000 7f570b800340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T11:55:58.967 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T11:55:58.943+0000 7f570b800340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T11:55:58.967 INFO:tasks.rook.operator.smithi119.stdout:couldn't init storage provider. : exit status 5 2022-04-09T11:55:58.967 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:55:58.960026 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T11:55:58.943+0000 7f570b800340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T11:55:58.967 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T11:55:58.943+0000 7f570b800340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T11:55:58.968 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T11:55:58.943+0000 7f570b800340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T11:55:58.968 INFO:tasks.rook.operator.smithi119.stdout:couldn't init storage provider. : exit status 5 2022-04-09T11:55:59.307 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:55:59.301799 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-04-09T11:55:59.308 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:55:59.301816 I | op-mds: successfully deleted mds config for "mds.foo-b" in mon configuration database 2022-04-09T11:55:59.308 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:55:59.301820 I | cephclient: deleting ceph auth "mds.foo-b" 2022-04-09T11:56:00.508 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:56:00.502540 I | op-mds: successfully deleted mds CephX key for "mds.foo-b" 2022-04-09T11:56:00.509 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:56:00.502643 I | ceph-file-controller: downing filesystem "foo" 2022-04-09T11:56:02.440 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:56:02.435243 I | ceph-file-controller: downed filesystem "foo" 2022-04-09T11:56:05.392 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:56:05.386861 I | cephclient: no images/snapshosts present in pool "foo-metadata" 2022-04-09T11:56:05.392 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:56:05.386883 I | cephclient: purging pool "foo-metadata" (id=9) 2022-04-09T11:56:06.929 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T11:56:07.754 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:56:07.748389 I | cephclient: purge completed for pool "foo-metadata" 2022-04-09T11:56:07.901 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:56:07.902 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:47:42.000000Z", "last_refresh": "2022-04-09T11:56:07.809658Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:56:07.809658Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T11:56:07.809658Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T11:56:07.809658Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T11:56:07.809658Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:56:07.809658Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:56:07.809658Z", "running": 0, "size": 1}}] 2022-04-09T11:56:08.168 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:56:08.162140 I | cephclient: no images/snapshosts present in pool "foo-data0" 2022-04-09T11:56:08.168 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:56:08.162168 I | cephclient: purging pool "foo-data0" (id=10) 2022-04-09T11:56:10.031 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:56:10.025058 I | cephclient: purge completed for pool "foo-data0" 2022-04-09T11:56:10.032 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:56:10.025103 I | ceph-spec: removing finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-04-09T11:56:10.037 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:56:10.031926 E | ceph-file-controller: failed to reconcile failed to remove finalizer: failed to remove finalizer "cephfilesystem.ceph.rook.io" on "foo": Operation cannot be fulfilled on cephfilesystems.ceph.rook.io "foo": the object has been modified; please apply your changes to the latest version and try again 2022-04-09T11:56:10.047 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:56:10.041555 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T11:56:10.841 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:56:10.834621 I | ceph-spec: removing finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-04-09T11:56:10.995 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:56:10.989848 I | ceph-spec: object "rook-ceph-mds-foo-b" matched on delete, reconciling 2022-04-09T11:56:10.996 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:56:10.990024 I | ceph-spec: object "rook-ceph-mds-foo-a" matched on delete, reconciling 2022-04-09T11:56:11.003 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:56:10.996311 I | ceph-spec: object "rook-ceph-mds-foo-b-keyring" matched on delete, reconciling 2022-04-09T11:56:11.003 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:56:10.996412 I | ceph-spec: object "rook-ceph-mds-foo-a-keyring" matched on delete, reconciling 2022-04-09T11:56:15.083 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:56:15.077383 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi119": the object has been modified; please apply your changes to the latest version and try again 2022-04-09T11:56:15.481 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 11:56:15.459761 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi119": the object has been modified; please apply your changes to the latest version and try again 2022-04-09T11:56:17.922 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T11:56:18.663 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:56:18.664 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:47:42.000000Z", "last_refresh": "2022-04-09T11:56:18.418869Z", "running": 2, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T11:56:18.418869Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T11:56:18.418869Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T11:56:18.418869Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:56:18.418869Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:56:18.418869Z", "running": 0, "size": 1}}] 2022-04-09T11:56:28.686 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T11:56:29.626 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:56:29.627 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:47:42.000000Z", "last_refresh": "2022-04-09T11:56:29.305379Z", "running": 2, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T11:56:29.305379Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T11:56:29.305379Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T11:56:29.305379Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:56:29.305379Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:56:29.305379Z", "running": 0, "size": 1}}] 2022-04-09T11:56:39.650 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T11:56:40.263 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:56:40.263 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T11:56:40.163763Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T11:56:40.163763Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T11:56:40.163763Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T11:56:40.163763Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:56:40.163763Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:56:40.163763Z", "running": 0, "size": 1}}] 2022-04-09T11:56:50.292 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T11:56:50.947 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:56:50.947 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T11:56:50.824134Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T11:56:50.824134Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T11:56:50.824134Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T11:56:50.824134Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:56:50.824134Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:56:50.824134Z", "running": 0, "size": 1}}] 2022-04-09T11:57:00.972 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T11:57:02.737 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:57:02.737 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T11:57:01.935078Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T11:57:01.935078Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T11:57:01.935078Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T11:57:01.935078Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:57:01.935078Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:57:01.935078Z", "running": 0, "size": 1}}] 2022-04-09T11:57:12.761 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T11:57:13.369 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:57:13.369 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T11:57:13.268017Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T11:57:13.268017Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T11:57:13.268017Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T11:57:13.268017Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:57:13.268017Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:57:13.268017Z", "running": 0, "size": 1}}] 2022-04-09T11:57:23.397 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T11:57:24.097 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:57:24.098 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T11:57:23.927204Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T11:57:23.927204Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T11:57:23.927204Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T11:57:23.927204Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:57:23.927204Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:57:23.927204Z", "running": 0, "size": 1}}] 2022-04-09T11:57:34.123 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T11:57:37.867 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:57:37.867 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T11:57:37.763409Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T11:57:37.763409Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T11:57:37.763409Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T11:57:37.763409Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:57:37.763409Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:57:37.763409Z", "running": 0, "size": 1}}] 2022-04-09T11:57:47.890 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T11:57:48.552 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:57:48.552 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T11:57:48.445703Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T11:57:48.445703Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T11:57:48.445703Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T11:57:48.445703Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:57:48.445703Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:57:48.445703Z", "running": 0, "size": 1}}] 2022-04-09T11:57:58.573 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T11:57:59.196 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:57:59.197 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T11:57:59.094190Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T11:57:59.094190Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T11:57:59.094190Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T11:57:59.094190Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:57:59.094190Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:57:59.094190Z", "running": 0, "size": 1}}] 2022-04-09T11:58:09.222 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T11:58:09.831 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:58:09.831 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T11:58:09.734260Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T11:58:09.734260Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T11:58:09.734260Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T11:58:09.734260Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:58:09.734260Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:58:09.734260Z", "running": 0, "size": 1}}] 2022-04-09T11:58:19.851 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T11:58:20.470 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:58:20.471 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T11:58:20.380843Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T11:58:20.380843Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T11:58:20.380843Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T11:58:20.380843Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:58:20.380843Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:58:20.380843Z", "running": 0, "size": 1}}] 2022-04-09T11:58:30.493 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T11:58:31.114 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:58:31.115 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T11:58:31.021271Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T11:58:31.021271Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T11:58:31.021271Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T11:58:31.021271Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:58:31.021271Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:58:31.021271Z", "running": 0, "size": 1}}] 2022-04-09T11:58:41.134 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T11:58:41.837 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:58:41.837 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T11:58:41.657750Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T11:58:41.657750Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T11:58:41.657750Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T11:58:41.657750Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:58:41.657750Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:58:41.657750Z", "running": 0, "size": 1}}] 2022-04-09T11:58:51.863 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T11:58:52.452 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:58:52.452 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T11:58:52.366118Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T11:58:52.366118Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T11:58:52.366118Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T11:58:52.366118Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:58:52.366118Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:58:52.366118Z", "running": 0, "size": 1}}] 2022-04-09T11:59:02.474 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T11:59:03.157 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:59:03.158 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T11:59:03.064510Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T11:59:03.064510Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T11:59:03.064510Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T11:59:03.064510Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:59:03.064510Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:59:03.064510Z", "running": 0, "size": 1}}] 2022-04-09T11:59:13.180 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T11:59:13.789 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:59:13.789 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T11:59:13.688167Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T11:59:13.688167Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T11:59:13.688167Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T11:59:13.688167Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:59:13.688167Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:59:13.688167Z", "running": 0, "size": 1}}] 2022-04-09T11:59:23.811 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T11:59:24.412 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:59:24.412 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T11:59:24.318649Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T11:59:24.318649Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T11:59:24.318649Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T11:59:24.318649Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:59:24.318649Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:59:24.318649Z", "running": 0, "size": 1}}] 2022-04-09T11:59:34.450 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T11:59:35.086 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:59:35.086 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T11:59:34.954656Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T11:59:34.954656Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T11:59:34.954656Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T11:59:34.954656Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:59:34.954656Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:59:34.954656Z", "running": 0, "size": 1}}] 2022-04-09T11:59:45.111 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T11:59:45.747 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:59:45.747 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T11:59:45.648390Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T11:59:45.648390Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T11:59:45.648390Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T11:59:45.648390Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:59:45.648390Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:59:45.648390Z", "running": 0, "size": 1}}] 2022-04-09T11:59:55.773 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T11:59:56.451 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T11:59:56.451 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T11:59:56.347548Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T11:59:56.347548Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T11:59:56.347548Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T11:59:56.347548Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T11:59:56.347548Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T11:59:56.347548Z", "running": 0, "size": 1}}] 2022-04-09T12:00:06.474 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:00:07.125 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:00:07.125 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:00:06.994584Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:00:06.994584Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:00:06.994584Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:00:06.994584Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:00:06.994584Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:00:06.994584Z", "running": 0, "size": 1}}] 2022-04-09T12:00:17.150 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:00:17.742 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:00:17.742 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:00:17.638144Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:00:17.638144Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:00:17.638144Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:00:17.638144Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:00:17.638144Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:00:17.638144Z", "running": 0, "size": 1}}] 2022-04-09T12:00:27.766 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:00:28.401 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:00:28.401 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:00:28.271262Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:00:28.271262Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:00:28.271262Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:00:28.271262Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:00:28.271262Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:00:28.271262Z", "running": 0, "size": 1}}] 2022-04-09T12:00:38.426 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:00:39.032 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:00:39.033 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:00:38.937214Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:00:38.937214Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:00:38.937214Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:00:38.937214Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:00:38.937214Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:00:38.937214Z", "running": 0, "size": 1}}] 2022-04-09T12:00:49.058 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:00:49.716 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:00:49.716 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:00:49.615238Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:00:49.615238Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:00:49.615238Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:00:49.615238Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:00:49.615238Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:00:49.615238Z", "running": 0, "size": 1}}] 2022-04-09T12:00:59.736 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:01:00.680 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:01:00.680 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:01:00.259533Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:01:00.259533Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:01:00.259533Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:01:00.259533Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:01:00.259533Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:01:00.259533Z", "running": 0, "size": 1}}] 2022-04-09T12:01:04.985 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:01:04.977951 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T12:01:05.584 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:01:05.576003 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T12:01:05.566+0000 7efffe1e9340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T12:01:05.585 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T12:01:05.566+0000 7efffe1e9340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T12:01:05.585 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T12:01:05.566+0000 7efffe1e9340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T12:01:05.586 INFO:tasks.rook.operator.smithi119.stdout:couldn't init storage provider. : exit status 5 2022-04-09T12:01:05.586 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:01:05.576025 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T12:01:05.566+0000 7efffe1e9340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T12:01:05.587 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T12:01:05.566+0000 7efffe1e9340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T12:01:05.587 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T12:01:05.566+0000 7efffe1e9340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T12:01:05.588 INFO:tasks.rook.operator.smithi119.stdout:couldn't init storage provider. : exit status 5 2022-04-09T12:01:10.702 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:01:11.325 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:01:11.326 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:01:11.176116Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:01:11.176116Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:01:11.176116Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:01:11.176116Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:01:11.176116Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:01:11.176116Z", "running": 0, "size": 1}}] 2022-04-09T12:01:21.350 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:01:22.020 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:01:22.020 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:01:21.887236Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:01:21.887236Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:01:21.887236Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:01:21.887236Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:01:21.887236Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:01:21.887236Z", "running": 0, "size": 1}}] 2022-04-09T12:01:32.042 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:01:32.691 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:01:32.692 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:01:32.569936Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:01:32.569936Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:01:32.569936Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:01:32.569936Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:01:32.569936Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:01:32.569936Z", "running": 0, "size": 1}}] 2022-04-09T12:01:42.713 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:01:43.374 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:01:43.374 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:01:43.266613Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:01:43.266613Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:01:43.266613Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:01:43.266613Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:01:43.266613Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:01:43.266613Z", "running": 0, "size": 1}}] 2022-04-09T12:01:53.396 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:01:54.052 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:01:54.052 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:01:53.914604Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:01:53.914604Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:01:53.914604Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:01:53.914604Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:01:53.914604Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:01:53.914604Z", "running": 0, "size": 1}}] 2022-04-09T12:02:04.077 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:02:04.704 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:02:04.704 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:02:04.605992Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:02:04.605992Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:02:04.605992Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:02:04.605992Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:02:04.605992Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:02:04.605992Z", "running": 0, "size": 1}}] 2022-04-09T12:02:14.728 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:02:15.346 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:02:15.346 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:02:15.245750Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:02:15.245750Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:02:15.245750Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:02:15.245750Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:02:15.245750Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:02:15.245750Z", "running": 0, "size": 1}}] 2022-04-09T12:02:25.370 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:02:26.016 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:02:26.016 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:02:25.862257Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:02:25.862257Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:02:25.862257Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:02:25.862257Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:02:25.862257Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:02:25.862257Z", "running": 0, "size": 1}}] 2022-04-09T12:02:36.036 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:02:36.666 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:02:36.666 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:02:36.567195Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:02:36.567195Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:02:36.567195Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:02:36.567195Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:02:36.567195Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:02:36.567195Z", "running": 0, "size": 1}}] 2022-04-09T12:02:46.687 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:02:47.319 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:02:47.320 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:02:47.202063Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:02:47.202063Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:02:47.202063Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:02:47.202063Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:02:47.202063Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:02:47.202063Z", "running": 0, "size": 1}}] 2022-04-09T12:02:57.344 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:02:57.968 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:02:57.969 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:02:57.863947Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:02:57.863947Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:02:57.863947Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:02:57.863947Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:02:57.863947Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:02:57.863947Z", "running": 0, "size": 1}}] 2022-04-09T12:03:07.993 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:03:08.628 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:03:08.628 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:03:08.527109Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:03:08.527109Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:03:08.527109Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:03:08.527109Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:03:08.527109Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:03:08.527109Z", "running": 0, "size": 1}}] 2022-04-09T12:03:18.652 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:03:19.412 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:03:19.412 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:03:19.214310Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:03:19.214310Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:03:19.214310Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:03:19.214310Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:03:19.214310Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:03:19.214310Z", "running": 0, "size": 1}}] 2022-04-09T12:03:29.436 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:03:30.029 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:03:30.029 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:03:29.939140Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:03:29.939140Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:03:29.939140Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:03:29.939140Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:03:29.939140Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:03:29.939140Z", "running": 0, "size": 1}}] 2022-04-09T12:03:40.052 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:03:40.647 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:03:40.647 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:03:40.542736Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:03:40.542736Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:03:40.542736Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:03:40.542736Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:03:40.542736Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:03:40.542736Z", "running": 0, "size": 1}}] 2022-04-09T12:03:50.668 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:03:51.294 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:03:51.294 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:03:51.176977Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:03:51.176977Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:03:51.176977Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:03:51.176977Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:03:51.176977Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:03:51.176977Z", "running": 0, "size": 1}}] 2022-04-09T12:04:01.316 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:04:02.089 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:04:02.090 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:04:01.904448Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:04:01.904448Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:04:01.904448Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:04:01.904448Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:04:01.904448Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:04:01.904448Z", "running": 0, "size": 1}}] 2022-04-09T12:04:12.111 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:04:12.760 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:04:12.760 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:04:12.632476Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:04:12.632476Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:04:12.632476Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:04:12.632476Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:04:12.632476Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:04:12.632476Z", "running": 0, "size": 1}}] 2022-04-09T12:04:22.781 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:04:23.351 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:04:23.351 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:04:23.252464Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:04:23.252464Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:04:23.252464Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:04:23.252464Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:04:23.252464Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:04:23.252464Z", "running": 0, "size": 1}}] 2022-04-09T12:04:33.376 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:04:33.987 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:04:33.988 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:04:33.882392Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:04:33.882392Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:04:33.882392Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:04:33.882392Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:04:33.882392Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:04:33.882392Z", "running": 0, "size": 1}}] 2022-04-09T12:04:44.010 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:04:44.644 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:04:44.644 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:04:44.540340Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:04:44.540340Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:04:44.540340Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:04:44.540340Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:04:44.540340Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:04:44.540340Z", "running": 0, "size": 1}}] 2022-04-09T12:04:54.663 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:04:55.227 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:04:55.228 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:04:55.137525Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:04:55.137525Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:04:55.137525Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:04:55.137525Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:04:55.137525Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:04:55.137525Z", "running": 0, "size": 1}}] 2022-04-09T12:05:05.250 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:05:05.874 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:05:05.875 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:05:05.776581Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:05:05.776581Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:05:05.776581Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:05:05.776581Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:05:05.776581Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:05:05.776581Z", "running": 0, "size": 1}}] 2022-04-09T12:05:15.898 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:05:16.507 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:05:16.507 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:05:16.411703Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:05:16.411703Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:05:16.411703Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:05:16.411703Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:05:16.411703Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:05:16.411703Z", "running": 0, "size": 1}}] 2022-04-09T12:05:26.530 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:05:27.135 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:05:27.136 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:05:27.035445Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:05:27.035445Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:05:27.035445Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:05:27.035445Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:05:27.035445Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:05:27.035445Z", "running": 0, "size": 1}}] 2022-04-09T12:05:37.160 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:05:37.766 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:05:37.767 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:05:37.664949Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:05:37.664949Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:05:37.664949Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:05:37.664949Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:05:37.664949Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:05:37.664949Z", "running": 0, "size": 1}}] 2022-04-09T12:05:47.789 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:05:48.402 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:05:48.402 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:05:48.303354Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:05:48.303354Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:05:48.303354Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:05:48.303354Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:05:48.303354Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:05:48.303354Z", "running": 0, "size": 1}}] 2022-04-09T12:05:58.424 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:05:59.104 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:05:59.105 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:05:58.956525Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:05:58.956525Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:05:58.956525Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:05:58.956525Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:05:58.956525Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:05:58.956525Z", "running": 0, "size": 1}}] 2022-04-09T12:06:09.132 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:06:09.779 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:06:09.779 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:06:09.682194Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:06:09.682194Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:06:09.682194Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:06:09.682194Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:06:09.682194Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:06:09.682194Z", "running": 0, "size": 1}}] 2022-04-09T12:06:19.802 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:06:20.533 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:06:20.533 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:06:20.441442Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:06:20.441442Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:06:20.441442Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:06:20.441442Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:06:20.441442Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:06:20.441442Z", "running": 0, "size": 1}}] 2022-04-09T12:06:30.559 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:06:31.183 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:06:31.183 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:06:31.085057Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:06:31.085057Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:06:31.085057Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:06:31.085057Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:06:31.085057Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:06:31.085057Z", "running": 0, "size": 1}}] 2022-04-09T12:06:41.207 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:06:41.872 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:06:41.872 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:06:41.770282Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:06:41.770282Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:06:41.770282Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:06:41.770282Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:06:41.770282Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:06:41.770282Z", "running": 0, "size": 1}}] 2022-04-09T12:06:51.895 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:06:52.590 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:06:52.591 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:06:52.482718Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:06:52.482718Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:06:52.482718Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:06:52.482718Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:06:52.482718Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:06:52.482718Z", "running": 0, "size": 1}}] 2022-04-09T12:07:02.615 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:07:03.236 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:07:03.236 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:07:03.123429Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:07:03.123429Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:07:03.123429Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:07:03.123429Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:07:03.123429Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:07:03.123429Z", "running": 0, "size": 1}}] 2022-04-09T12:07:13.260 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:07:13.850 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:07:13.850 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:07:13.748783Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:07:13.748783Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:07:13.748783Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:07:13.748783Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:07:13.748783Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:07:13.748783Z", "running": 0, "size": 1}}] 2022-04-09T12:07:23.871 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:07:24.467 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:07:24.467 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:07:24.373409Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:07:24.373409Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:07:24.373409Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:07:24.373409Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:07:24.373409Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:07:24.373409Z", "running": 0, "size": 1}}] 2022-04-09T12:07:34.489 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:07:35.206 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:07:35.207 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:07:34.996158Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:07:34.996158Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:07:34.996158Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:07:34.996158Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:07:34.996158Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:07:34.996158Z", "running": 0, "size": 1}}] 2022-04-09T12:07:45.230 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:07:48.647 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:07:48.648 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:07:48.542426Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:07:48.542426Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:07:48.542426Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:07:48.542426Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:07:48.542426Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:07:48.542426Z", "running": 0, "size": 1}}] 2022-04-09T12:07:58.671 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:07:59.311 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:07:59.312 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:07:59.217131Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:07:59.217131Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:07:59.217131Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:07:59.217131Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:07:59.217131Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:07:59.217131Z", "running": 0, "size": 1}}] 2022-04-09T12:08:09.332 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:08:09.976 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:08:09.977 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:08:09.852743Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:08:09.852743Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:08:09.852743Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:08:09.852743Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:08:09.852743Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:08:09.852743Z", "running": 0, "size": 1}}] 2022-04-09T12:08:20.000 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:08:20.616 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:08:20.616 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:08:20.524984Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:08:20.524984Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:08:20.524984Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:08:20.524984Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:08:20.524984Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:08:20.524984Z", "running": 0, "size": 1}}] 2022-04-09T12:08:30.636 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:08:31.281 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:08:31.282 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:08:31.154903Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:08:31.154903Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:08:31.154903Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:08:31.154903Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:08:31.154903Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:08:31.154903Z", "running": 0, "size": 1}}] 2022-04-09T12:08:41.306 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:08:41.976 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:08:41.976 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:08:41.877123Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:08:41.877123Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:08:41.877123Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:08:41.877123Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:08:41.877123Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:08:41.877123Z", "running": 0, "size": 1}}] 2022-04-09T12:08:51.998 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:08:52.612 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:08:52.613 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:08:52.525838Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:08:52.525838Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:08:52.525838Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:08:52.525838Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:08:52.525838Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:08:52.525838Z", "running": 0, "size": 1}}] 2022-04-09T12:09:02.635 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:09:03.266 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:09:03.266 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:09:03.166420Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:09:03.166420Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:09:03.166420Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:09:03.166420Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:09:03.166420Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:09:03.166420Z", "running": 0, "size": 1}}] 2022-04-09T12:09:13.288 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:09:13.904 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:09:13.904 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:09:13.807186Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:09:13.807186Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:09:13.807186Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:09:13.807186Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:09:13.807186Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:09:13.807186Z", "running": 0, "size": 1}}] 2022-04-09T12:09:23.927 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:09:24.635 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:09:24.635 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:09:24.514024Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:09:24.514024Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:09:24.514024Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:09:24.514024Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:09:24.514024Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:09:24.514024Z", "running": 0, "size": 1}}] 2022-04-09T12:09:34.658 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:09:35.269 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:09:35.269 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:09:35.173076Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:09:35.173076Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:09:35.173076Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:09:35.173076Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:09:35.173076Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:09:35.173076Z", "running": 0, "size": 1}}] 2022-04-09T12:09:45.291 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:09:45.901 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:09:45.902 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:09:45.807173Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:09:45.807173Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:09:45.807173Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:09:45.807173Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:09:45.807173Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:09:45.807173Z", "running": 0, "size": 1}}] 2022-04-09T12:09:55.928 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:09:56.590 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:09:56.591 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:09:56.444337Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:09:56.444337Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:09:56.444337Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:09:56.444337Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:09:56.444337Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:09:56.444337Z", "running": 0, "size": 1}}] 2022-04-09T12:10:06.613 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:10:07.292 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:10:07.292 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:10:07.148989Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:10:07.148989Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:10:07.148989Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:10:07.148989Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:10:07.148989Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:10:07.148989Z", "running": 0, "size": 1}}] 2022-04-09T12:10:17.317 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:10:17.942 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:10:17.942 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:10:17.843272Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:10:17.843272Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:10:17.843272Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:10:17.843272Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:10:17.843272Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:10:17.843272Z", "running": 0, "size": 1}}] 2022-04-09T12:10:27.968 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:10:28.578 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:10:28.579 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:10:28.478831Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:10:28.478831Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:10:28.478831Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:10:28.478831Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:10:28.478831Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:10:28.478831Z", "running": 0, "size": 1}}] 2022-04-09T12:10:38.603 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:10:39.247 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:10:39.248 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:10:39.109941Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:10:39.109941Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:10:39.109941Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:10:39.109941Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:10:39.109941Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:10:39.109941Z", "running": 0, "size": 1}}] 2022-04-09T12:10:49.270 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:10:49.893 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:10:49.893 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:10:49.797103Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:10:49.797103Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:10:49.797103Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:10:49.797103Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:10:49.797103Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:10:49.797103Z", "running": 0, "size": 1}}] 2022-04-09T12:10:59.917 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:11:00.588 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:11:00.589 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:11:00.421220Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:11:00.421220Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:11:00.421220Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:11:00.421220Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:11:00.421220Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:11:00.421220Z", "running": 0, "size": 1}}] 2022-04-09T12:11:10.615 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:11:11.205 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:11:11.205 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:11:11.105115Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:11:11.105115Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:11:11.105115Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:11:11.105115Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:11:11.105115Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:11:11.105115Z", "running": 0, "size": 1}}] 2022-04-09T12:11:21.229 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:11:21.880 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:11:21.880 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:11:21.759097Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:11:21.759097Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:11:21.759097Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:11:21.759097Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:11:21.759097Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:11:21.759097Z", "running": 0, "size": 1}}] 2022-04-09T12:11:31.904 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:11:32.639 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:11:32.639 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:11:32.398599Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:11:32.398599Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:11:32.398599Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:11:32.398599Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:11:32.398599Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:11:32.398599Z", "running": 0, "size": 1}}] 2022-04-09T12:11:42.658 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:11:43.294 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:11:43.295 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:11:43.202235Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:11:43.202235Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:11:43.202235Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:11:43.202235Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:11:43.202235Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:11:43.202235Z", "running": 0, "size": 1}}] 2022-04-09T12:11:53.320 DEBUG:teuthology.orchestra.run.smithi119:> kubectl -n rook-ceph exec rook-ceph-tools-7677f7f4b9-69kwt -- ceph orch ls -f json 2022-04-09T12:11:53.916 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-04-09T12:11:53.916 INFO:teuthology.orchestra.run.smithi119.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:56:15.000000Z", "last_refresh": "2022-04-09T12:11:53.819108Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:58.000000Z", "last_refresh": "2022-04-09T12:11:53.819108Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:37:31.000000Z", "last_refresh": "2022-04-09T12:11:53.819108Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2abf022dd4de4425e43021e8e2346ef594551dec1ca2739aea51daba75b86a95", "container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "created": "2022-04-09T11:39:32.000000Z", "last_refresh": "2022-04-09T12:11:53.819108Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-04-09T12:11:53.819108Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:a70e525ea98f7194c5e516757c6105ca14c81cb7", "last_refresh": "2022-04-09T12:11:53.819108Z", "running": 0, "size": 1}}] 2022-04-09T12:11:53.937 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_a347b2ac5400a43ba76a678399b1deda7dae1a6c/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_a70e525ea98f7194c5e516757c6105ca14c81cb7/qa/tasks/rook.py", line 669, in task File "/home/teuthworker/src/git.ceph.com_git_teuthology_a347b2ac5400a43ba76a678399b1deda7dae1a6c/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-04-09T12:11:53.938 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_a70e525ea98f7194c5e516757c6105ca14c81cb7/qa/tasks/rook.py", line 530, in ceph_config_keyring File "/home/teuthworker/src/git.ceph.com_git_teuthology_a347b2ac5400a43ba76a678399b1deda7dae1a6c/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_a70e525ea98f7194c5e516757c6105ca14c81cb7/qa/tasks/rook.py", line 669, in task File "/home/teuthworker/src/git.ceph.com_git_teuthology_a347b2ac5400a43ba76a678399b1deda7dae1a6c/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-04-09T12:11:53.939 INFO:tasks.rook:Cleaning up config and client.admin keyring 2022-04-09T12:11:53.939 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-04-09T12:11:54.029 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_a70e525ea98f7194c5e516757c6105ca14c81cb7/qa/tasks/rook.py", line 478, in rook_post_config File "/home/teuthworker/src/git.ceph.com_git_teuthology_a347b2ac5400a43ba76a678399b1deda7dae1a6c/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_a70e525ea98f7194c5e516757c6105ca14c81cb7/qa/tasks/rook.py", line 669, in task File "/home/teuthworker/src/git.ceph.com_git_teuthology_a347b2ac5400a43ba76a678399b1deda7dae1a6c/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-04-09T12:11:54.029 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_a70e525ea98f7194c5e516757c6105ca14c81cb7/qa/tasks/rook.py", line 442, in rook_toolbox File "/home/teuthworker/src/git.ceph.com_git_teuthology_a347b2ac5400a43ba76a678399b1deda7dae1a6c/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_a70e525ea98f7194c5e516757c6105ca14c81cb7/qa/tasks/rook.py", line 669, in task File "/home/teuthworker/src/git.ceph.com_git_teuthology_a347b2ac5400a43ba76a678399b1deda7dae1a6c/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-04-09T12:11:54.049 DEBUG:teuthology.orchestra.remote:smithi119:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB 2022-04-09T12:11:54.061 DEBUG:teuthology.orchestra.run.smithi119:> kubectl delete -f rook/cluster/examples/kubernetes/ceph/toolbox.yaml 2022-04-09T12:11:54.126 INFO:teuthology.orchestra.run.smithi119.stdout:deployment.apps "rook-ceph-tools" deleted 2022-04-09T12:11:54.135 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_a70e525ea98f7194c5e516757c6105ca14c81cb7/qa/tasks/rook.py", line 379, in rook_cluster File "/home/teuthworker/src/git.ceph.com_git_teuthology_a347b2ac5400a43ba76a678399b1deda7dae1a6c/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_a70e525ea98f7194c5e516757c6105ca14c81cb7/qa/tasks/rook.py", line 669, in task File "/home/teuthworker/src/git.ceph.com_git_teuthology_a347b2ac5400a43ba76a678399b1deda7dae1a6c/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-04-09T12:11:54.136 DEBUG:teuthology.orchestra.run.smithi119:> kubectl delete -f cluster.yaml 2022-04-09T12:11:54.340 INFO:teuthology.orchestra.run.smithi119.stdout:cephcluster.ceph.rook.io "rook-ceph" deleted 2022-04-09T12:11:54.343 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:11:54.335832 I | ceph-cluster-controller: CR "rook-ceph" is going be deleted, cancelling any ongoing orchestration 2022-04-09T12:11:54.718 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:11:54.712452 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:11:54.726 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:11:54.720943 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:11:54.727 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:11:54.720967 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:12:04.949 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:12:04.943368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:12:04.961 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:12:04.955480 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:12:15.184 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:12:15.178458 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:12:15.198 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:12:15.192453 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:12:25.428 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:12:25.422174 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:12:25.439 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:12:25.433666 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:12:35.668 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:12:35.662613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:12:35.678 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:12:35.672269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:12:45.899 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:12:45.893105 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:12:45.965 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:12:45.957301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:12:56.236 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:12:56.230655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:12:56.250 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:12:56.244552 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:13:06.487 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:13:06.481181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:13:06.500 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:13:06.494708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:13:16.726 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:13:16.720380 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:13:16.739 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:13:16.733958 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:13:26.989 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:13:26.983099 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:13:27.003 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:13:26.997392 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:13:37.226 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:13:37.220072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:13:37.238 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:13:37.232336 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:13:47.460 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:13:47.454258 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:13:47.471 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:13:47.466172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:13:57.693 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:13:57.687174 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:13:57.707 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:13:57.701733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:14:07.926 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:14:07.920546 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:14:07.940 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:14:07.934885 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:14:18.163 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:14:18.157669 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:14:18.204 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:14:18.199003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:14:28.427 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:14:28.421550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:14:28.436 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:14:28.430508 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:14:38.656 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:14:38.650552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:14:38.667 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:14:38.662263 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:14:48.885 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:14:48.879785 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:14:48.919 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:14:48.893893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:14:59.120 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:14:59.114977 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:14:59.133 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:14:59.127599 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:15:09.355 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:15:09.350268 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:15:09.364 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:15:09.359660 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:15:19.654 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:15:19.648897 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:15:19.665 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:15:19.660684 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:15:29.879 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:15:29.873632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:15:29.891 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:15:29.886338 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:15:40.109 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:15:40.104529 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:15:40.120 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:15:40.116166 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:15:50.343 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:15:50.338389 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:15:50.355 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:15:50.350150 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:16:00.574 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:16:00.569436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:16:00.681 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:16:00.583000 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:16:10.873 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:16:10.868868 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:16:10.883 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:16:10.879036 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:16:21.155 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:16:21.150475 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:16:21.167 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:16:21.162989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:16:31.391 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:16:31.387122 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:16:31.405 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:16:31.400796 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:16:41.651 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:16:41.647167 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:16:41.680 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:16:41.661494 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:16:51.882 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:16:51.877955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:16:51.891 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:16:51.887511 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:17:02.140 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:17:02.135478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:17:02.152 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:17:02.148250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:17:12.374 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:17:12.370025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:17:12.387 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:17:12.382437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:17:22.607 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:17:22.602413 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:17:22.620 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:17:22.615892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:17:32.846 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:17:32.842047 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:17:32.859 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:17:32.855155 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:17:43.078 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:17:43.074200 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:17:43.090 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:17:43.086127 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:17:45.588 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:17:45.583851 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T12:17:46.172 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:17:46.168027 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T12:17:46.161+0000 7f77fa00c340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T12:17:46.173 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T12:17:46.161+0000 7f77fa00c340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T12:17:46.173 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T12:17:46.161+0000 7f77fa00c340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T12:17:46.173 INFO:tasks.rook.operator.smithi119.stdout:couldn't init storage provider. : exit status 5 2022-04-09T12:17:46.174 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:17:46.168056 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T12:17:46.161+0000 7f77fa00c340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T12:17:46.174 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T12:17:46.161+0000 7f77fa00c340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T12:17:46.174 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T12:17:46.161+0000 7f77fa00c340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T12:17:46.174 INFO:tasks.rook.operator.smithi119.stdout:couldn't init storage provider. : exit status 5 2022-04-09T12:17:53.306 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:17:53.302224 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:17:53.318 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:17:53.314127 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:18:03.539 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:18:03.535626 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:18:03.552 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:18:03.548917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:18:13.771 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:18:13.766690 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:18:14.249 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:18:13.779786 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:18:24.007 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:18:24.002254 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:18:24.021 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:18:24.017043 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:18:34.245 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:18:34.240653 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:18:34.257 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:18:34.252545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:18:44.500 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:18:44.496217 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:18:44.515 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:18:44.508208 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:18:54.726 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:18:54.722678 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:18:54.774 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:18:54.770735 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:19:04.995 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:19:04.991259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:19:05.007 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:19:05.004267 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:19:15.228 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:19:15.224224 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:19:15.240 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:19:15.236361 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:19:25.460 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:19:25.456211 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:19:25.470 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:19:25.466787 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:19:35.699 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:19:35.694694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:19:35.711 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:19:35.707311 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:19:45.930 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:19:45.926250 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:19:46.003 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:19:45.938700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:19:56.159 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:19:56.155745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:19:56.172 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:19:56.168690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:20:06.391 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:20:06.387978 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:20:06.438 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:20:06.401206 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:20:06.610 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:20:06.607029 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-04-09 12:20:06.597067806 +0000 UTC m=+2633.957967360 LastTransitionTime:2022-04-09 11:38:08 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-04-09T12:20:16.627 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:20:16.623052 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:20:16.639 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:20:16.635083 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:20:26.865 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:20:26.861464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:20:26.877 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:20:26.874022 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:20:37.100 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:20:37.096392 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:20:37.111 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:20:37.108108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:20:47.336 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:20:47.332932 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:20:47.351 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:20:47.346446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:20:57.575 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:20:57.571851 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:20:57.620 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:20:57.616677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:21:07.839 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:21:07.835875 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:21:07.851 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:21:07.847529 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:21:18.096 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:21:18.092520 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:21:18.124 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:21:18.120969 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:21:28.362 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:21:28.358931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:21:28.375 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:21:28.371904 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:21:38.625 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:21:38.621603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:21:38.704 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:21:38.633069 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:21:48.857 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:21:48.853595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:21:48.897 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:21:48.893151 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:21:59.117 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:21:59.113296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:21:59.128 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:21:59.125057 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:22:09.346 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:22:09.342889 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:22:09.358 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:22:09.355238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:22:19.576 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:22:19.572327 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:22:19.588 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:22:19.585450 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:22:29.811 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:22:29.808215 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:22:29.888 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:22:29.883577 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:22:40.102 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:22:40.098851 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:22:40.115 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:22:40.111824 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:22:50.374 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:22:50.371185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:22:50.384 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:22:50.381020 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:23:00.690 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:23:00.686261 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:23:00.702 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:23:00.699311 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:23:10.922 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:23:10.918370 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:23:10.934 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:23:10.931271 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:23:21.155 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:23:21.151799 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:23:21.168 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:23:21.165391 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:23:31.409 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:23:31.406478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:23:31.422 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:23:31.419697 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:23:41.641 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:23:41.638493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:23:41.662 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:23:41.658867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:23:51.881 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:23:51.877820 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:23:51.892 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:23:51.889170 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:24:02.119 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:24:02.116087 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:24:02.132 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:24:02.128620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:24:12.353 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:24:12.350405 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:24:12.366 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:24:12.363413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:24:22.623 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:24:22.620059 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:24:22.639 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:24:22.633001 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:24:32.856 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:24:32.853205 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:24:32.871 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:24:32.868170 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:24:43.095 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:24:43.092118 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:24:43.106 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:24:43.103688 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:24:53.329 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:24:53.326202 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:24:53.342 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:24:53.339424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:25:03.586 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:25:03.583126 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:25:03.598 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:25:03.595688 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:25:13.817 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:25:13.813515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:25:13.828 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:25:13.825269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:25:24.065 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:25:24.062706 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:25:24.074 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:25:24.071615 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:25:34.288 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:25:34.285593 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:25:34.302 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:25:34.298440 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:25:44.515 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:25:44.512598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:25:44.524 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:25:44.521522 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:25:54.801 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:25:54.798502 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:25:54.812 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:25:54.809692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:26:05.035 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:26:05.031826 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:26:05.047 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:26:05.043879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:26:15.269 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:26:15.265893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:26:15.283 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:26:15.280593 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:26:25.522 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:26:25.519248 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:26:25.531 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:26:25.528419 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:26:35.747 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:26:35.744041 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:26:35.762 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:26:35.759177 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:26:45.983 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:26:45.980123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:26:45.996 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:26:45.993781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:26:56.216 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:26:56.213716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:26:56.228 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:26:56.223206 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:27:06.451 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:27:06.448521 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:27:06.459 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:27:06.456808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:27:16.706 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:27:16.703169 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:27:16.774 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:27:16.770993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:27:26.992 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:27:26.988841 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:27:27.004 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:27:27.001792 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:27:37.221 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:27:37.218405 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:27:37.235 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:27:37.232788 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:27:47.452 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:27:47.448960 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:27:47.464 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:27:47.461498 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:27:57.683 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:27:57.680443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:27:57.697 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:27:57.694375 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:28:07.923 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:28:07.920063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:28:07.936 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:28:07.933876 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:28:18.163 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:28:18.160488 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:28:18.175 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:28:18.172491 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:28:28.397 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:28:28.394085 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:28:28.412 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:28:28.408172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:28:38.636 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:28:38.633517 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:28:38.649 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:28:38.646339 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:28:48.874 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:28:48.870743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:28:48.887 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:28:48.884410 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:28:59.139 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:28:59.135979 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:28:59.150 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:28:59.147955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:29:09.377 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:29:09.374066 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:29:09.389 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:29:09.386711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:29:19.627 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:29:19.624140 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:29:19.662 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:29:19.659035 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:29:29.877 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:29:29.874305 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:29:29.890 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:29:29.887529 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:29:40.155 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:29:40.152142 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:29:40.167 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:29:40.164550 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:29:50.440 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:29:50.436825 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:29:50.453 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:29:50.450101 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:30:00.675 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:30:00.672600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:30:00.688 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:30:00.685495 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:30:10.909 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:30:10.906365 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:30:10.921 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:30:10.919104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:30:21.147 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:30:21.144070 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:30:21.160 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:30:21.157210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:30:31.395 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:30:31.392086 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:30:31.440 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:30:31.437384 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:30:41.660 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:30:41.657300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:30:41.670 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:30:41.667514 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:30:51.893 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:30:51.889823 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:30:51.905 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:30:51.903204 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:31:02.129 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:31:02.125362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:31:02.140 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:31:02.137836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:31:12.358 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:31:12.355772 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:31:12.369 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:31:12.366539 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:31:22.588 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:31:22.585058 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:31:22.601 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:31:22.598485 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:31:32.847 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:31:32.844079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:31:32.859 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:31:32.856425 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:31:43.095 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:31:43.092276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:31:43.106 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:31:43.103204 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:31:53.325 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:31:53.322049 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:31:53.338 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:31:53.335569 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:32:03.558 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:32:03.555184 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:32:03.568 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:32:03.565792 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:32:13.783 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:32:13.779972 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:32:13.795 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:32:13.792457 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:32:24.020 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:32:24.016965 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:32:24.030 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:32:24.027571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:32:34.253 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:32:34.250467 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:32:34.265 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:32:34.262147 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:32:44.491 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:32:44.488026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:32:44.505 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:32:44.502603 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:32:54.732 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:32:54.729403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:32:54.743 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:32:54.740388 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:33:04.998 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:33:04.995466 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:33:05.012 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:33:05.009146 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:33:15.255 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:33:15.251922 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:33:15.266 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:33:15.263773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:33:25.519 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:33:25.516475 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:33:25.531 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:33:25.529001 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:33:35.750 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:33:35.747086 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:33:35.763 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:33:35.760012 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:33:46.000 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:33:45.996991 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:33:46.010 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:33:46.007658 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:33:56.292 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:33:56.289435 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:33:56.304 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:33:56.301432 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:34:06.529 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:34:06.526553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:34:06.539 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:34:06.536414 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:34:16.796 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:34:16.793239 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:34:16.807 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:34:16.804483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:34:26.179 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:34:26.175829 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T12:34:26.857 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:34:26.853768 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T12:34:26.845+0000 7fef3adb4340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T12:34:26.857 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T12:34:26.845+0000 7fef3adb4340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T12:34:26.858 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T12:34:26.845+0000 7fef3adb4340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T12:34:26.858 INFO:tasks.rook.operator.smithi119.stdout:couldn't init storage provider. : exit status 5 2022-04-09T12:34:26.858 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:34:26.853789 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T12:34:26.845+0000 7fef3adb4340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T12:34:26.858 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T12:34:26.845+0000 7fef3adb4340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T12:34:26.859 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T12:34:26.845+0000 7fef3adb4340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T12:34:26.859 INFO:tasks.rook.operator.smithi119.stdout:couldn't init storage provider. : exit status 5 2022-04-09T12:34:27.027 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:34:27.023791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:34:27.037 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:34:27.035198 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:34:37.291 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:34:37.288248 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:34:37.303 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:34:37.300281 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:34:47.521 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:34:47.517822 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:34:47.530 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:34:47.527472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:34:57.760 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:34:57.757370 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:34:57.849 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:34:57.846658 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:35:08.067 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:35:08.063790 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:35:08.078 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:35:08.074920 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:35:18.297 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:35:18.293638 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:35:18.309 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:35:18.306800 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:35:28.530 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:35:28.526761 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:35:28.542 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:35:28.539533 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:35:38.773 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:35:38.769991 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:35:38.785 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:35:38.781861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:35:49.010 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:35:49.006351 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:35:49.036 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:35:49.033371 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:35:59.256 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:35:59.252549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:35:59.267 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:35:59.264145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:36:09.487 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:36:09.484189 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:36:09.499 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:36:09.495773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:36:19.719 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:36:19.716454 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:36:19.730 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:36:19.727145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:36:29.957 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:36:29.953562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:36:29.969 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:36:29.966286 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:36:40.195 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:36:40.192197 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:36:40.209 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:36:40.206404 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:36:50.432 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:36:50.429229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:36:50.444 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:36:50.440966 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:37:00.674 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:37:00.670649 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:37:00.685 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:37:00.682068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:37:10.909 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:37:10.906264 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:37:10.921 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:37:10.918740 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:37:21.196 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:37:21.192742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:37:21.212 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:37:21.209378 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:37:21.277 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:37:21.274792 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-04-09 12:37:21.262805238 +0000 UTC m=+3668.623704750 LastTransitionTime:2022-04-09 11:38:08 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-04-09T12:37:31.441 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:37:31.437500 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:37:31.453 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:37:31.449752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:37:41.671 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:37:41.667977 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:37:41.683 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:37:41.679482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:37:51.909 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:37:51.905996 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:37:51.920 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:37:51.917252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:38:02.144 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:38:02.140885 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:38:02.153 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:38:02.150196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:38:12.375 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:38:12.371887 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:38:12.388 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:38:12.385305 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:38:22.606 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:38:22.602814 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:38:22.639 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:38:22.636005 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:38:32.859 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:38:32.855301 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:38:32.873 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:38:32.869403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:38:43.126 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:38:43.122286 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:38:43.134 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:38:43.131702 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:38:53.368 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:38:53.364610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:38:53.377 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:38:53.373657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:39:03.592 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:39:03.589256 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:39:03.602 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:39:03.599305 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:39:13.880 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:39:13.876571 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:39:13.891 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:39:13.888364 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:39:24.114 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:39:24.110043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:39:24.123 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:39:24.120259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:39:34.346 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:39:34.342859 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:39:34.358 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:39:34.354879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:39:44.616 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:39:44.612319 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:39:44.625 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:39:44.622226 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:39:54.848 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:39:54.844012 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:39:54.859 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:39:54.855881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:40:05.083 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:40:05.079512 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:40:05.094 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:40:05.090985 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:40:15.315 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:40:15.311528 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:40:15.326 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:40:15.323622 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:40:25.549 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:40:25.545134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:40:25.560 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:40:25.556380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:40:35.782 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:40:35.777685 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:40:35.797 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:40:35.794267 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:40:46.017 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:40:46.013730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:40:46.028 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:40:46.024846 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:40:56.251 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:40:56.247180 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:40:56.262 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:40:56.258776 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:41:06.505 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:41:06.501588 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:41:06.515 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:41:06.511990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:41:16.736 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:41:16.732246 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:41:16.748 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:41:16.744500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:41:26.966 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:41:26.962557 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:41:26.982 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:41:26.977017 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:41:37.206 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:41:37.201877 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:41:37.219 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:41:37.215527 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:41:47.439 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:41:47.435295 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:41:47.476 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:41:47.472186 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:41:57.745 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:41:57.740366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:41:57.753 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:41:57.749893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:42:07.976 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:42:07.972802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:42:07.987 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:42:07.983897 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:42:18.211 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:42:18.207921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:42:18.220 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:42:18.217235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:42:28.448 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:42:28.444434 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:42:28.459 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:42:28.455183 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:42:38.679 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:42:38.674850 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:42:38.690 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:42:38.686904 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:42:48.913 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:42:48.908912 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:42:48.925 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:42:48.921353 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:42:59.150 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:42:59.146179 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:42:59.163 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:42:59.159417 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:43:09.379 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:43:09.375064 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:43:09.391 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:43:09.387168 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:43:19.612 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:43:19.608102 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:43:19.624 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:43:19.620021 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:43:29.843 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:43:29.839147 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:43:29.852 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:43:29.849055 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:43:40.071 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:43:40.067047 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:43:40.084 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:43:40.080072 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:43:50.347 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:43:50.343330 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:43:50.361 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:43:50.357287 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:44:00.590 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:44:00.585679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:44:00.604 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:44:00.600672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:44:10.827 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:44:10.822665 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:44:10.838 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:44:10.834284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:44:21.201 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:44:21.196946 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:44:21.216 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:44:21.210835 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:44:31.437 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:44:31.432957 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:44:31.447 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:44:31.443084 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:44:41.669 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:44:41.664515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:44:41.678 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:44:41.674217 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:44:52.290 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:44:52.285691 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:44:52.300 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:44:52.295917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:45:02.531 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:45:02.526918 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:45:02.542 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:45:02.538713 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:45:12.767 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:45:12.762642 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:45:12.779 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:45:12.774892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:45:23.268 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:45:23.263659 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:45:23.279 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:45:23.275092 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:45:33.525 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:45:33.521231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:45:33.538 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:45:33.534395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:45:43.761 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:45:43.757453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:45:43.775 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:45:43.770707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:45:53.999 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:45:53.994636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:45:54.008 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:45:54.004426 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:46:04.224 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:46:04.219972 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:46:04.236 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:46:04.232174 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:46:14.468 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:46:14.464144 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:46:14.479 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:46:14.475305 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:46:24.697 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:46:24.692880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:46:24.713 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:46:24.708934 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:46:34.933 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:46:34.929208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:46:34.946 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:46:34.940781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:46:45.165 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:46:45.160817 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:46:45.182 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:46:45.177776 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:46:55.404 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:46:55.399755 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:46:55.414 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:46:55.410160 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:47:05.636 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:47:05.631264 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:47:05.648 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:47:05.643898 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:47:15.870 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:47:15.865634 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:47:15.882 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:47:15.877236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:47:26.114 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:47:26.109771 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:47:26.125 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:47:26.121352 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:47:36.379 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:47:36.373775 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:47:36.394 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:47:36.390107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:47:46.615 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:47:46.610449 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:47:46.665 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:47:46.660970 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:47:56.885 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:47:56.881043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:47:56.897 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:47:56.893475 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:48:07.121 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:48:07.117049 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:48:07.132 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:48:07.127829 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:48:17.356 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:48:17.351246 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:48:17.368 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:48:17.364330 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:48:27.595 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:48:27.590761 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:48:27.607 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:48:27.602762 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:48:37.836 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:48:37.831776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:48:37.846 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:48:37.841502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:48:48.068 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:48:48.063892 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:48:48.081 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:48:48.076715 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:48:58.305 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:48:58.300476 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:48:58.317 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:48:58.312730 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:49:08.537 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:49:08.532651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:49:08.552 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:49:08.547976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:49:18.774 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:49:18.769565 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:49:18.785 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:49:18.781116 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:49:29.077 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:49:29.073087 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:49:29.085 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:49:29.080788 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:49:39.306 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:49:39.301879 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:49:39.319 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:49:39.315443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:49:49.540 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:49:49.535768 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:49:49.551 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:49:49.547635 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:49:59.872 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:49:59.867351 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:49:59.884 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:49:59.879733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:50:10.105 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:50:10.100409 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:50:10.118 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:50:10.113680 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:50:20.355 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:50:20.350839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:50:20.367 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:50:20.362928 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:50:30.641 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:50:30.636803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:50:30.653 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:50:30.649247 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:50:40.873 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:50:40.868432 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:50:40.886 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:50:40.882045 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:50:51.107 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:50:51.102976 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:50:51.120 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:50:51.116116 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:51:01.407 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:51:01.402376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:51:01.421 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:51:01.416447 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:51:06.868 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:51:06.863990 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T12:51:07.478 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:51:07.472886 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T12:51:07.462+0000 7ff173e0d340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T12:51:07.478 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T12:51:07.462+0000 7ff173e0d340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T12:51:07.478 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T12:51:07.462+0000 7ff173e0d340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T12:51:07.479 INFO:tasks.rook.operator.smithi119.stdout:couldn't init storage provider. : exit status 5 2022-04-09T12:51:07.479 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:51:07.472910 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T12:51:07.462+0000 7ff173e0d340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T12:51:07.480 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T12:51:07.462+0000 7ff173e0d340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T12:51:07.480 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T12:51:07.462+0000 7ff173e0d340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T12:51:07.480 INFO:tasks.rook.operator.smithi119.stdout:couldn't init storage provider. : exit status 5 2022-04-09T12:51:11.643 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:51:11.638999 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:51:11.653 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:51:11.647672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:51:21.872 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:51:21.867817 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:51:21.883 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:51:21.879059 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:51:32.101 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:51:32.096849 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:51:32.115 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:51:32.110638 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:51:42.344 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:51:42.340036 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:51:42.354 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:51:42.350411 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:51:52.580 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:51:52.575376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:51:52.597 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:51:52.592711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:52:02.818 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:52:02.813169 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:52:02.829 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:52:02.825266 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:52:13.084 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:52:13.079797 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:52:13.093 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:52:13.089379 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:52:23.316 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:52:23.311202 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:52:23.323 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:52:23.319466 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:52:33.543 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:52:33.538115 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:52:33.553 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:52:33.548863 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:52:43.817 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:52:43.812380 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:52:43.835 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:52:43.830872 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:52:54.085 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:52:54.079909 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:52:54.098 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:52:54.093884 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:53:04.318 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:53:04.312974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:53:04.329 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:53:04.325111 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:53:14.549 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:53:14.544168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:53:14.561 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:53:14.557040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:53:24.780 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:53:24.775368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:53:24.793 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:53:24.788485 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:53:35.011 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:53:35.006862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:53:35.022 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:53:35.017922 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:53:35.170 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:53:35.165849 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-04-09 12:53:35.155143307 +0000 UTC m=+4642.516042844 LastTransitionTime:2022-04-09 11:38:08 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-04-09T12:53:45.239 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:53:45.234506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:53:45.251 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:53:45.246301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:53:55.476 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:53:55.470737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:53:55.488 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:53:55.483734 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:54:05.748 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:54:05.742758 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:54:05.757 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:54:05.753196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:54:15.977 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:54:15.971483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:54:15.986 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:54:15.981636 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:54:26.239 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:54:26.233895 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:54:26.247 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:54:26.242400 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:54:36.618 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:54:36.613287 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:54:36.629 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:54:36.624976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:54:46.850 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:54:46.845642 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:54:46.861 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:54:46.856543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:54:57.099 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:54:57.094091 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:54:57.112 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:54:57.107701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:55:07.333 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:55:07.327698 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:55:07.345 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:55:07.340697 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:55:17.596 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:55:17.591137 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:55:17.607 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:55:17.602664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:55:27.829 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:55:27.824535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:55:27.838 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:55:27.833793 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:55:38.216 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:55:38.211207 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:55:38.222 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:55:38.218210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:55:48.442 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:55:48.436705 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:55:48.454 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:55:48.449011 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:55:58.674 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:55:58.668824 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:55:58.688 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:55:58.683807 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:56:08.913 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:56:08.907509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:56:08.927 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:56:08.922970 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:56:19.150 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:56:19.145293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:56:19.251 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:56:19.246045 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:56:29.468 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:56:29.463584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:56:29.519 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:56:29.514701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:56:39.935 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:56:39.930421 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:56:39.945 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:56:39.940178 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:56:50.193 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:56:50.188245 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:56:50.206 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:56:50.201121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:57:00.424 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:57:00.419147 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:57:00.439 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:57:00.434448 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:57:10.656 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:57:10.650788 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:57:10.725 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:57:10.720067 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:57:20.946 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:57:20.941466 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:57:20.958 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:57:20.953152 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:57:31.181 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:57:31.176133 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:57:31.193 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:57:31.188058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:57:41.473 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:57:41.467552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:57:41.558 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:57:41.552659 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:57:51.832 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:57:51.826823 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:57:51.842 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:57:51.837536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:58:02.059 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:58:02.053843 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:58:02.071 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:58:02.065953 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:58:12.292 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:58:12.287161 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:58:12.303 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:58:12.299017 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:58:22.527 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:58:22.521201 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:58:22.538 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:58:22.533771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:58:32.761 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:58:32.756308 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:58:32.773 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:58:32.768211 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:58:43.152 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:58:43.146663 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:58:43.163 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:58:43.157661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:58:53.383 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:58:53.378322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:58:53.461 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:58:53.456759 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:59:03.684 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:59:03.678547 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:59:03.695 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:59:03.690016 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:59:13.945 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:59:13.939956 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:59:13.955 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:59:13.950076 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:59:24.177 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:59:24.172000 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:59:24.190 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:59:24.185347 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:59:34.415 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:59:34.409406 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:59:34.426 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:59:34.420641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:59:44.689 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:59:44.684242 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:59:44.701 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:59:44.696011 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:59:54.929 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:59:54.924291 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T12:59:54.995 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 12:59:54.989860 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:00:05.247 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:00:05.241512 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:00:05.258 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:00:05.253183 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:00:15.633 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:00:15.627041 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:00:15.641 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:00:15.636757 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:00:25.860 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:00:25.854989 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:00:25.872 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:00:25.867633 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:00:36.091 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:00:36.085504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:00:36.100 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:00:36.095360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:00:46.322 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:00:46.317440 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:00:46.334 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:00:46.329717 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:00:56.557 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:00:56.551053 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:00:56.568 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:00:56.563003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:01:06.788 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:01:06.782651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:01:06.801 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:01:06.796046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:01:17.023 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:01:17.018454 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:01:17.034 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:01:17.029341 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:01:27.277 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:01:27.271227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:01:27.293 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:01:27.287518 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:01:37.574 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:01:37.567984 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:01:37.588 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:01:37.582954 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:01:47.831 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:01:47.825490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:01:47.844 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:01:47.839157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:01:58.064 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:01:58.058403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:01:58.076 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:01:58.070902 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:02:08.296 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:02:08.290920 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:02:08.309 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:02:08.303772 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:02:18.581 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:02:18.576296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:02:18.617 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:02:18.612012 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:02:28.848 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:02:28.842879 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:02:28.859 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:02:28.854476 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:02:39.084 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:02:39.079078 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:02:39.098 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:02:39.093342 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:02:49.342 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:02:49.336387 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:02:49.351 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:02:49.346072 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:02:59.626 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:02:59.621317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:02:59.638 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:02:59.633159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:03:09.858 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:03:09.852963 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:03:09.871 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:03:09.865783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:03:20.098 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:03:20.092436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:03:20.113 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:03:20.107669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:03:30.334 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:03:30.328651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:03:30.345 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:03:30.340395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:03:40.570 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:03:40.564891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:03:40.583 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:03:40.577427 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:03:50.931 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:03:50.925316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:03:50.945 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:03:50.939800 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:04:01.162 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:04:01.156025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:04:01.175 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:04:01.169467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:04:11.393 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:04:11.387939 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:04:11.405 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:04:11.400390 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:04:21.676 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:04:21.670689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:04:21.686 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:04:21.681571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:04:31.908 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:04:31.902736 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:04:31.919 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:04:31.913602 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:04:42.141 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:04:42.135344 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:04:42.155 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:04:42.149537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:04:52.475 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:04:52.469285 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:04:52.485 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:04:52.480364 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:05:02.732 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:05:02.726872 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:05:02.744 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:05:02.739665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:05:12.960 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:05:12.954835 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:05:12.972 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:05:12.967403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:05:23.302 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:05:23.296131 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:05:23.314 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:05:23.309015 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:05:33.526 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:05:33.520611 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:05:33.539 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:05:33.533820 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:05:43.758 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:05:43.752766 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:05:43.769 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:05:43.764437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:05:54.175 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:05:54.169653 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:05:54.186 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:05:54.181116 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:06:04.409 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:06:04.403155 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:06:04.421 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:06:04.415675 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:06:14.679 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:06:14.673291 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:06:14.689 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:06:14.683509 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:06:24.911 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:06:24.905584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:06:24.925 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:06:24.919684 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:06:35.151 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:06:35.144975 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:06:35.163 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:06:35.157511 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:06:45.579 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:06:45.573827 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:06:45.589 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:06:45.584192 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:06:55.815 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:06:55.809006 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:06:55.827 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:06:55.821430 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:07:06.043 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:07:06.037696 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:07:06.058 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:07:06.052656 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:07:16.335 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:07:16.329948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:07:16.348 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:07:16.342352 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:07:26.793 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:07:26.787141 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:07:26.823 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:07:26.817775 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:07:37.090 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:07:37.084737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:07:37.104 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:07:37.098668 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:07:47.219 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:07:47.213497 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-04-09 13:07:47.204340295 +0000 UTC m=+5494.565239892 LastTransitionTime:2022-04-09 11:38:08 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-04-09T13:07:47.321 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:07:47.315482 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:07:47.331 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:07:47.325252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:07:47.484 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:07:47.479150 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T13:07:48.096 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:07:48.089721 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T13:07:48.080+0000 7f6c5e3b1340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T13:07:48.096 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T13:07:48.080+0000 7f6c5e3b1340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T13:07:48.096 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T13:07:48.080+0000 7f6c5e3b1340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T13:07:48.097 INFO:tasks.rook.operator.smithi119.stdout:couldn't init storage provider. : exit status 5 2022-04-09T13:07:48.097 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:07:48.089748 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T13:07:48.080+0000 7f6c5e3b1340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T13:07:48.097 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T13:07:48.080+0000 7f6c5e3b1340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T13:07:48.097 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T13:07:48.080+0000 7f6c5e3b1340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T13:07:48.098 INFO:tasks.rook.operator.smithi119.stdout:couldn't init storage provider. : exit status 5 2022-04-09T13:07:57.552 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:07:57.546104 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:07:57.562 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:07:57.557395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:08:07.784 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:08:07.777985 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:08:07.795 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:08:07.789919 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:08:18.021 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:08:18.015446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:08:18.032 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:08:18.027266 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:08:28.295 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:08:28.288979 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:08:28.570 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:08:28.564874 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:08:38.816 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:08:38.810542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:08:38.828 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:08:38.822112 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:08:49.051 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:08:49.044854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:08:49.062 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:08:49.056755 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:08:59.281 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:08:59.275599 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:08:59.292 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:08:59.286636 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:09:09.516 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:09:09.510588 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:09:09.529 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:09:09.523780 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:09:19.750 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:09:19.744230 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:09:19.762 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:09:19.756635 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:09:29.983 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:09:29.976818 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:09:29.996 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:09:29.990858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:09:40.218 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:09:40.211766 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:09:40.229 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:09:40.223707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:09:50.452 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:09:50.445552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:09:50.462 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:09:50.456634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:10:00.683 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:10:00.677431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:10:00.697 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:10:00.691702 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:10:10.924 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:10:10.917663 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:10:10.934 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:10:10.928905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:10:21.158 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:10:21.151443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:10:21.168 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:10:21.162765 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:10:31.394 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:10:31.387490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:10:31.404 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:10:31.398485 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:10:41.661 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:10:41.655023 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:10:41.674 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:10:41.668960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:10:51.900 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:10:51.894340 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:10:51.966 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:10:51.959831 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:11:02.230 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:11:02.224474 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:11:02.242 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:11:02.236463 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:11:12.465 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:11:12.458520 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:11:12.478 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:11:12.472309 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:11:22.698 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:11:22.692038 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:11:22.708 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:11:22.701936 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:11:32.936 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:11:32.930195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:11:32.949 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:11:32.943233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:11:43.190 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:11:43.184051 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:11:43.203 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:11:43.197512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:11:53.423 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:11:53.417350 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:11:53.437 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:11:53.431306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:12:03.661 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:12:03.655156 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:12:03.674 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:12:03.668122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:12:03.674 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:12:03.668152 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:12:13.894 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:12:13.888412 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:12:13.906 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:12:13.900675 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:12:24.128 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:12:24.121812 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:12:24.141 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:12:24.134989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:12:34.359 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:12:34.352256 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:12:34.369 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:12:34.363413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:12:44.623 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:12:44.617113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:12:44.635 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:12:44.629270 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:12:54.855 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:12:54.848787 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:12:54.877 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:12:54.869853 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:13:05.098 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:13:05.091731 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:13:05.111 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:13:05.104999 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:13:15.354 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:13:15.347399 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:13:15.365 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:13:15.359256 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:13:25.624 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:13:25.618209 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:13:25.635 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:13:25.629205 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:13:35.862 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:13:35.855890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:13:35.877 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:13:35.870965 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:13:46.100 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:13:46.093326 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:13:46.113 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:13:46.106435 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:13:56.336 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:13:56.330068 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:13:56.350 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:13:56.343850 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:14:06.577 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:14:06.570516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:14:06.591 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:14:06.585166 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:14:16.812 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:14:16.806473 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:14:16.825 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:14:16.819199 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:14:27.054 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:14:27.047761 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:14:27.066 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:14:27.060001 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:14:37.298 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:14:37.291295 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:14:37.309 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:14:37.303373 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:14:47.530 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:14:47.523721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:14:47.541 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:14:47.535470 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:14:57.769 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:14:57.762406 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:14:57.781 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:14:57.775532 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:15:08.008 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:15:08.001056 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:15:08.021 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:15:08.015369 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:15:18.245 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:15:18.238672 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:15:18.255 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:15:18.249621 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:15:28.475 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:15:28.468534 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:15:28.492 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:15:28.484914 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:15:38.736 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:15:38.729601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:15:38.751 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:15:38.744486 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:15:48.969 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:15:48.963227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:15:48.983 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:15:48.976856 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:15:59.207 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:15:59.200694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:15:59.222 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:15:59.215797 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:16:09.444 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:16:09.437239 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:16:09.462 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:16:09.455064 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:16:19.686 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:16:19.679068 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:16:19.700 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:16:19.693719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:16:29.919 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:16:29.912808 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:16:29.933 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:16:29.926532 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:16:40.167 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:16:40.160551 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:16:40.181 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:16:40.175197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:16:50.401 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:16:50.394307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:16:50.414 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:16:50.408133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:17:00.639 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:17:00.632196 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:17:00.650 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:17:00.643913 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:17:10.877 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:17:10.863646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:17:10.882 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:17:10.876342 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:17:21.111 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:17:21.104063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:17:21.124 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:17:21.117588 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:17:31.375 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:17:31.368022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:17:31.386 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:17:31.379434 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:17:41.624 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:17:41.617636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:17:41.635 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:17:41.629171 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:17:51.856 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:17:51.849014 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:17:51.867 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:17:51.861277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:18:02.089 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:18:02.082310 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:18:02.102 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:18:02.095685 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:18:12.324 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:18:12.316852 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:18:12.335 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:18:12.328459 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:18:22.561 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:18:22.553657 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:18:22.571 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:18:22.564630 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:18:32.786 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:18:32.779780 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:18:32.799 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:18:32.792400 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:18:43.021 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:18:43.013851 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:18:43.032 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:18:43.025600 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:18:53.252 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:18:53.245667 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:18:53.265 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:18:53.258993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:19:03.493 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:19:03.485829 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:19:03.507 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:19:03.500798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:19:13.728 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:19:13.720982 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:19:13.739 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:19:13.732726 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:19:24.001 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:19:23.993862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:19:24.097 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:19:24.090819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:19:34.319 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:19:34.311575 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:19:34.327 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:19:34.320794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:19:44.575 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:19:44.568508 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:19:44.585 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:19:44.578123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:19:54.803 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:19:54.796298 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:19:54.816 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:19:54.808300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:20:05.034 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:20:05.026945 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:20:05.046 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:20:05.039411 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:20:15.267 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:20:15.260051 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:20:15.278 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:20:15.269614 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:20:25.494 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:20:25.486764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:20:25.506 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:20:25.499136 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:20:35.725 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:20:35.717631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:20:35.738 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:20:35.731880 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:20:45.956 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:20:45.949372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:20:45.991 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:20:45.984066 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:20:56.213 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:20:56.205366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:20:56.250 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:20:56.242862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:21:06.483 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:21:06.475652 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:21:06.495 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:21:06.488009 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:21:16.722 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:21:16.715057 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:21:16.736 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:21:16.728669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:21:26.960 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:21:26.952847 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:21:26.972 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:21:26.965146 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:21:37.204 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:21:37.197434 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:21:37.216 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:21:37.209685 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:21:47.441 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:21:47.433714 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:21:47.502 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:21:47.495282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:21:57.732 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:21:57.724737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:21:57.741 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:21:57.734376 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:22:07.965 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:22:07.957816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:22:07.977 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:22:07.970060 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:22:18.202 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:22:18.194932 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:22:18.215 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:22:18.208594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:22:28.443 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:22:28.435276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:22:28.454 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:22:28.447202 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:22:38.682 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:22:38.674309 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:22:38.694 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:22:38.687059 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:22:48.913 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:22:48.906046 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:22:48.925 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:22:48.918329 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:22:59.152 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:22:59.145367 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:22:59.164 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:22:59.157049 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:23:09.391 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:23:09.383508 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:23:09.403 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:23:09.395923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:23:19.627 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:23:19.619389 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:23:19.641 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:23:19.634292 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:23:29.859 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:23:29.851600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:23:29.872 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:23:29.864987 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:23:40.115 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:23:40.107751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:23:40.128 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:23:40.121434 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:23:50.346 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:23:50.338659 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:23:50.359 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:23:50.351636 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:24:00.576 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:24:00.569706 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:24:00.588 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:24:00.581218 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:24:10.811 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:24:10.804136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:24:10.820 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:24:10.813997 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:24:21.043 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:24:21.035663 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:24:21.054 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:24:21.047394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:24:28.109 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:24:28.102432 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T13:24:28.734 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:24:28.726297 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T13:24:28.717+0000 7fd1e94d4340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T13:24:28.735 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T13:24:28.717+0000 7fd1e94d4340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T13:24:28.735 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T13:24:28.717+0000 7fd1e94d4340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T13:24:28.735 INFO:tasks.rook.operator.smithi119.stdout:couldn't init storage provider. : exit status 5 2022-04-09T13:24:28.736 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:24:28.726322 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T13:24:28.717+0000 7fd1e94d4340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T13:24:28.736 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T13:24:28.717+0000 7fd1e94d4340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T13:24:28.736 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T13:24:28.717+0000 7fd1e94d4340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T13:24:28.736 INFO:tasks.rook.operator.smithi119.stdout:couldn't init storage provider. : exit status 5 2022-04-09T13:24:31.313 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:24:31.306604 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:24:31.355 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:24:31.347291 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:24:41.574 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:24:41.567176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:24:41.586 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:24:41.579576 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:24:51.807 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:24:51.799732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:24:51.818 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:24:51.811814 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:25:02.035 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:25:02.028695 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:25:02.047 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:25:02.040797 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:25:02.079 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:25:02.072280 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-04-09 13:25:02.04899323 +0000 UTC m=+6529.409892735 LastTransitionTime:2022-04-09 11:38:08 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-04-09T13:25:12.271 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:25:12.263911 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:25:12.283 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:25:12.276942 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:25:22.507 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:25:22.500600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:25:22.518 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:25:22.511366 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:25:32.741 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:25:32.735003 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:25:32.749 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:25:32.743040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:25:42.972 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:25:42.965181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:25:42.980 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:25:42.974171 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:25:53.209 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:25:53.202624 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:25:53.218 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:25:53.212357 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:26:03.452 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:26:03.445114 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:26:03.461 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:26:03.454182 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:26:13.681 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:26:13.673940 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:26:13.691 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:26:13.684371 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:26:23.906 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:26:23.899553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:26:23.919 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:26:23.912273 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:26:34.138 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:26:34.131020 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:26:34.157 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:26:34.150362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:26:44.383 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:26:44.376710 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:26:44.393 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:26:44.386673 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:26:54.618 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:26:54.611336 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:26:54.629 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:26:54.622741 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:27:04.849 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:27:04.841661 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:27:04.860 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:27:04.853331 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:27:15.084 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:27:15.077110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:27:15.096 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:27:15.089812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:27:25.327 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:27:25.320488 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:27:25.341 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:27:25.334178 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:27:35.558 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:27:35.551462 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:27:35.571 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:27:35.565086 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:27:45.808 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:27:45.800908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:27:45.851 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:27:45.844626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:27:56.098 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:27:56.091314 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:27:56.116 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:27:56.107039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:28:06.330 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:28:06.323937 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:28:06.343 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:28:06.336675 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:28:16.564 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:28:16.557872 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:28:16.574 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:28:16.568328 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:28:26.794 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:28:26.787893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:28:26.806 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:28:26.800151 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:28:37.028 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:28:37.021757 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:28:37.046 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:28:37.040279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:28:47.273 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:28:47.267196 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:28:47.329 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:28:47.321432 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:28:57.545 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:28:57.538739 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:28:57.559 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:28:57.553552 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:29:07.782 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:29:07.775913 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:29:07.796 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:29:07.789620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:29:18.020 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:29:18.010606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:29:18.032 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:29:18.025741 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:29:28.253 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:29:28.246684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:29:28.267 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:29:28.260227 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:29:38.489 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:29:38.482958 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:29:38.501 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:29:38.494999 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:29:48.729 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:29:48.722466 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:29:48.741 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:29:48.735480 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:29:58.958 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:29:58.951910 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:29:58.973 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:29:58.966903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:30:09.196 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:30:09.189750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:30:09.214 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:30:09.208089 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:30:19.440 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:30:19.433431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:30:19.452 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:30:19.446254 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:30:29.689 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:30:29.682856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:30:29.701 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:30:29.695091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:30:39.918 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:30:39.911817 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:30:39.925 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:30:39.919159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:30:50.150 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:30:50.143517 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:30:50.165 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:30:50.159049 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:31:00.419 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:31:00.413286 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:31:00.432 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:31:00.425514 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:31:10.655 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:31:10.648881 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:31:10.672 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:31:10.664863 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:31:20.899 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:31:20.892495 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:31:20.909 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:31:20.903164 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:31:31.145 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:31:31.139552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:31:31.161 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:31:31.155216 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:31:41.417 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:31:41.411382 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:31:41.430 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:31:41.423647 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:31:51.650 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:31:51.643861 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:31:51.664 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:31:51.658440 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:32:01.884 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:32:01.878527 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:32:01.897 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:32:01.891403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:32:12.118 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:32:12.112175 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:32:12.129 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:32:12.123786 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:32:22.350 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:32:22.344269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:32:22.361 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:32:22.355693 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:32:32.610 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:32:32.604062 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:32:32.682 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:32:32.676206 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:32:42.949 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:32:42.943228 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:32:42.964 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:32:42.958191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:32:53.193 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:32:53.187535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:32:53.205 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:32:53.199310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:33:03.455 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:33:03.449512 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:33:03.464 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:33:03.458913 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:33:13.687 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:33:13.680737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:33:13.699 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:33:13.692903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:33:23.921 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:33:23.914762 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:33:23.932 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:33:23.926525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:33:34.153 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:33:34.147020 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:33:34.165 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:33:34.159093 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:33:44.384 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:33:44.377515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:33:44.394 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:33:44.388760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:33:54.616 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:33:54.609568 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:33:54.627 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:33:54.621710 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:34:04.850 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:34:04.844036 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:34:04.863 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:34:04.857134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:34:15.085 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:34:15.078475 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:34:15.098 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:34:15.092542 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:34:25.320 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:34:25.314711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:34:25.330 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:34:25.324193 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:34:35.559 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:34:35.553299 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:34:35.570 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:34:35.564702 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:34:45.791 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:34:45.784593 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:34:45.818 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:34:45.812061 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:34:56.042 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:34:56.035721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:34:56.054 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:34:56.048705 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:35:06.284 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:35:06.278171 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:35:06.345 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:35:06.339338 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:35:16.582 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:35:16.575784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:35:16.594 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:35:16.588938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:35:26.819 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:35:26.812831 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:35:26.829 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:35:26.823355 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:35:37.061 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:35:37.055301 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:35:37.073 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:35:37.067544 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:35:47.290 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:35:47.284687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:35:47.301 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:35:47.295532 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:35:57.545 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:35:57.538686 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:35:57.556 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:35:57.551063 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:36:07.777 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:36:07.771425 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:36:07.792 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:36:07.785860 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:36:18.014 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:36:18.008205 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:36:18.027 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:36:18.021595 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:36:28.248 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:36:28.241799 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:36:28.261 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:36:28.254774 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:36:38.486 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:36:38.479947 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:36:38.497 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:36:38.491614 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:36:48.711 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:36:48.705520 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:36:48.722 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:36:48.715324 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:36:58.962 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:36:58.956093 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:36:58.975 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:36:58.969003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:37:09.196 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:37:09.190066 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:37:09.208 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:37:09.202868 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:37:19.428 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:37:19.422235 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:37:19.441 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:37:19.435615 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:37:29.662 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:37:29.656162 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:37:29.675 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:37:29.669387 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:37:39.989 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:37:39.983089 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:37:40.003 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:37:39.997136 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:37:50.221 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:37:50.215383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:37:50.233 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:37:50.227504 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:38:00.482 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:38:00.475690 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:38:00.497 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:38:00.491249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:38:10.718 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:38:10.711635 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:38:10.728 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:38:10.722144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:38:20.944 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:38:20.937960 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:38:20.957 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:38:20.951097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:38:31.173 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:38:31.167618 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:38:31.186 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:38:31.179701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:38:41.408 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:38:41.402047 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:38:41.422 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:38:41.417117 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:38:51.644 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:38:51.638415 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:38:51.662 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:38:51.656945 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:39:01.884 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:39:01.878783 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:39:01.897 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:39:01.891578 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:39:12.121 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:39:12.114974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:39:12.133 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:39:12.127606 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:39:22.356 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:39:22.350558 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:39:22.370 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:39:22.364246 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:39:32.599 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:39:32.593796 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:39:32.624 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:39:32.618876 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:39:42.920 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:39:42.914799 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:39:42.932 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:39:42.926898 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:39:53.155 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:39:53.149323 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:39:53.195 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:39:53.190076 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:40:03.417 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:40:03.411502 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:40:03.428 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:40:03.422334 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:40:13.647 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:40:13.641283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:40:13.659 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:40:13.653288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:40:23.881 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:40:23.875253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:40:23.893 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:40:23.888345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:40:34.115 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:40:34.109466 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:40:34.128 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:40:34.122636 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:40:44.347 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:40:44.341219 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:40:44.414 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:40:44.408654 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:40:54.643 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:40:54.637888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:40:54.656 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:40:54.650515 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:41:04.870 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:41:04.864261 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:41:04.882 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:41:04.876565 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:41:08.739 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:41:08.733989 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T13:41:09.511 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:41:09.505515 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T13:41:09.498+0000 7f44850e2340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T13:41:09.512 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T13:41:09.498+0000 7f44850e2340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T13:41:09.512 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T13:41:09.498+0000 7f44850e2340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T13:41:09.512 INFO:tasks.rook.operator.smithi119.stdout:couldn't init storage provider. : exit status 5 2022-04-09T13:41:09.512 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:41:09.505540 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T13:41:09.498+0000 7f44850e2340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T13:41:09.513 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T13:41:09.498+0000 7f44850e2340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T13:41:09.513 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T13:41:09.498+0000 7f44850e2340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T13:41:09.513 INFO:tasks.rook.operator.smithi119.stdout:couldn't init storage provider. : exit status 5 2022-04-09T13:41:15.098 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:41:15.092741 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:41:15.111 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:41:15.106031 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:41:25.333 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:41:25.327594 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:41:25.347 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:41:25.341539 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:41:35.570 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:41:35.564866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:41:35.584 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:41:35.578282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:41:45.811 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:41:45.805510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:41:45.821 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:41:45.816120 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:41:56.045 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:41:56.039351 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:41:56.057 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:41:56.051842 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:42:06.277 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:42:06.271950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:42:06.301 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:42:06.296748 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:42:16.520 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:42:16.513652 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:42:16.528 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:42:16.523277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:42:16.912 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:42:16.906909 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-04-09 13:42:16.893904908 +0000 UTC m=+7564.254804474 LastTransitionTime:2022-04-09 11:38:08 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-04-09T13:42:26.748 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:42:26.742806 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:42:26.758 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:42:26.753380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:42:36.982 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:42:36.976578 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:42:36.994 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:42:36.989151 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:42:47.218 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:42:47.212651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:42:47.231 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:42:47.226236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:42:57.454 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:42:57.448956 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:42:57.469 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:42:57.463893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:43:07.693 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:43:07.687678 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:43:07.707 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:43:07.701995 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:43:17.764 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:43:17.759324 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-04-09 13:43:17.725692809 +0000 UTC m=+7625.086592320 LastTransitionTime:2022-04-09 11:38:08 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-04-09T13:43:17.925 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:43:17.916121 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:43:17.936 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:43:17.931606 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:43:28.166 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:43:28.160783 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:43:28.178 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:43:28.173205 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:43:38.403 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:43:38.397581 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:43:38.415 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:43:38.409844 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:43:49.061 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:43:49.056317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:43:49.071 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:43:49.066795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:43:59.347 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:43:59.341647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:43:59.361 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:43:59.355567 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:44:09.582 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:44:09.577294 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:44:09.594 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:44:09.589446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:44:20.216 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:44:20.211391 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:44:20.227 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:44:20.223029 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:44:30.459 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:44:30.453814 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:44:30.468 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:44:30.463957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:44:40.739 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:44:40.734100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:44:40.754 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:44:40.749195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:44:51.087 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:44:51.081942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:44:51.098 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:44:51.093696 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:45:01.320 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:45:01.315069 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:45:01.333 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:45:01.328805 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:45:11.556 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:45:11.551307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:45:11.568 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:45:11.564137 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:45:21.792 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:45:21.787719 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:45:21.805 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:45:21.799442 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:45:32.036 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:45:32.031097 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:45:32.045 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:45:32.040448 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:45:42.292 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:45:42.286840 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:45:42.304 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:45:42.299299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:45:52.535 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:45:52.522928 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:45:52.539 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:45:52.534678 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:46:02.758 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:46:02.753652 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:46:02.772 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:46:02.767468 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:46:12.994 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:46:12.990025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:46:13.007 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:46:13.002981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:46:23.227 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:46:23.222661 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:46:23.239 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:46:23.235055 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:46:33.455 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:46:33.450662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:46:33.468 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:46:33.463917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:46:43.691 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:46:43.686481 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:46:43.702 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:46:43.698259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:46:53.923 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:46:53.918820 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:46:53.935 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:46:53.930217 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:47:04.157 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:47:04.152668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:47:04.169 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:47:04.164617 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:47:14.389 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:47:14.384458 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:47:14.401 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:47:14.396582 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:47:24.625 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:47:24.620487 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:47:24.636 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:47:24.631909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:47:34.870 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:47:34.865734 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:47:34.880 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:47:34.875844 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:47:45.122 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:47:45.117785 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:47:45.135 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:47:45.131170 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:47:55.375 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:47:55.370879 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:47:55.387 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:47:55.382662 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:48:05.608 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:48:05.603469 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:48:05.655 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:48:05.650519 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:48:15.873 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:48:15.868874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:48:15.885 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:48:15.880827 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:48:26.104 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:48:26.100031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:48:26.122 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:48:26.113437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:48:36.342 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:48:36.337825 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:48:36.355 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:48:36.351525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:48:46.580 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:48:46.575568 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:48:46.594 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:48:46.590509 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:48:56.810 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:48:56.805736 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:48:56.821 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:48:56.817496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:49:07.044 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:49:07.039591 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:49:07.057 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:49:07.053441 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:49:17.278 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:49:17.274328 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:49:17.288 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:49:17.284043 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:49:27.513 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:49:27.509036 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:49:27.527 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:49:27.522625 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:49:37.748 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:49:37.744226 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:49:37.760 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:49:37.756727 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:49:47.987 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:49:47.982980 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:49:48.002 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:49:47.997940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:49:58.263 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:49:58.259479 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:49:58.299 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:49:58.295517 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:50:08.554 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:50:08.550141 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:50:08.571 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:50:08.566111 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:50:18.794 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:50:18.790480 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:50:18.806 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:50:18.802676 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:50:29.029 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:50:29.025465 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:50:29.043 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:50:29.038293 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:50:39.266 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:50:39.262100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:50:39.277 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:50:39.273393 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:50:49.497 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:50:49.492727 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:50:49.509 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:50:49.504920 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:50:59.733 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:50:59.728752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:50:59.747 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:50:59.743014 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:51:09.979 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:51:09.974620 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:51:09.991 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:51:09.987140 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:51:20.211 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:51:20.207249 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:51:20.226 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:51:20.222258 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:51:30.482 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:51:30.477514 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:51:30.495 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:51:30.490791 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:51:40.716 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:51:40.711553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:51:40.756 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:51:40.752040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:51:51.003 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:51:50.999138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:51:51.018 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:51:51.013417 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:52:01.268 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:52:01.263940 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:52:01.281 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:52:01.277741 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:52:11.531 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:52:11.527176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:52:11.544 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:52:11.540365 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:52:21.763 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:52:21.759483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:52:21.777 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:52:21.773254 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:52:32.001 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:52:31.994060 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:52:32.018 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:52:32.009015 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:52:42.232 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:52:42.228495 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:52:42.246 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:52:42.242444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:52:52.466 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:52:52.461765 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:52:52.477 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:52:52.473648 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:53:02.703 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:53:02.699499 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:53:02.715 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:53:02.711783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:53:12.952 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:53:12.948098 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:53:12.963 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:53:12.959999 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:53:23.185 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:53:23.181796 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:53:23.521 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:53:23.193300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:53:33.417 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:53:33.413726 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:53:33.431 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:53:33.426772 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:53:43.653 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:53:43.649302 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:53:43.665 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:53:43.661528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:53:53.887 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:53:53.883055 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:53:53.899 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:53:53.895625 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:54:04.126 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:54:04.122108 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:54:04.141 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:54:04.135989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:54:14.360 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:54:14.355854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:54:14.372 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:54:14.368826 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:54:24.595 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:54:24.591544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:54:24.607 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:54:24.602993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:54:34.832 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:54:34.827934 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:54:34.875 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:54:34.871590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:54:45.099 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:54:45.095590 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:54:45.108 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:54:45.104306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:54:55.330 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:54:55.325263 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:54:55.341 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:54:55.336698 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:55:05.568 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:55:05.564400 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:55:05.580 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:55:05.576422 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:55:15.801 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:55:15.797053 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:55:15.813 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:55:15.810153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:55:26.042 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:55:26.038933 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:55:26.054 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:55:26.050393 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:55:36.283 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:55:36.278342 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:55:36.292 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:55:36.288510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:55:46.537 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:55:46.533871 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:55:46.551 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:55:46.547092 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:55:56.771 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:55:56.767248 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:55:56.784 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:55:56.780370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:56:07.004 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:56:07.000372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:56:07.016 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:56:07.012768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:56:17.250 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:56:17.246540 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:56:17.263 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:56:17.260039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:56:27.483 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:56:27.479876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:56:27.496 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:56:27.492888 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:56:37.716 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:56:37.712500 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:56:37.726 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:56:37.722789 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:56:47.949 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:56:47.945825 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:56:47.962 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:56:47.959352 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:56:58.186 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:56:58.183102 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:56:58.196 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:56:58.192913 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:57:08.431 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:57:08.427976 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:57:08.445 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:57:08.442052 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:57:18.666 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:57:18.662980 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:57:18.679 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:57:18.675963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:57:28.900 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:57:28.896155 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:57:28.912 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:57:28.907661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:57:39.136 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:57:39.132083 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:57:39.146 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:57:39.143292 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:57:49.370 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:57:49.366619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:57:49.382 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:57:49.379402 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:57:49.516 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:57:49.512936 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T13:57:50.160 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:57:50.155863 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T13:57:50.147+0000 7f423d824340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T13:57:50.160 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T13:57:50.147+0000 7f423d824340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T13:57:50.160 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T13:57:50.147+0000 7f423d824340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T13:57:50.161 INFO:tasks.rook.operator.smithi119.stdout:couldn't init storage provider. : exit status 5 2022-04-09T13:57:50.161 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:57:50.155892 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T13:57:50.147+0000 7f423d824340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T13:57:50.161 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T13:57:50.147+0000 7f423d824340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T13:57:50.161 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T13:57:50.147+0000 7f423d824340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T13:57:50.162 INFO:tasks.rook.operator.smithi119.stdout:couldn't init storage provider. : exit status 5 2022-04-09T13:57:59.601 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:57:59.597757 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:57:59.611 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:57:59.608553 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:58:09.860 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:58:09.856624 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:58:09.872 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:58:09.868406 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:58:20.097 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:58:20.093217 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:58:20.107 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:58:20.104014 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:58:30.329 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:58:30.325239 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:58:30.340 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:58:30.336632 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:58:40.565 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:58:40.561679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:58:40.574 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:58:40.570747 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:58:50.801 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:58:50.797861 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:58:50.813 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:58:50.809616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:59:01.040 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:59:01.036584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:59:01.051 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:59:01.048445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:59:11.280 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:59:11.276265 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:59:11.291 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:59:11.288489 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:59:21.520 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:59:21.517084 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:59:21.531 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:59:21.527682 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:59:31.747 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:59:31.743415 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:59:31.755 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:59:31.752467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:59:31.781 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:59:31.777966 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-04-09 13:59:31.757126668 +0000 UTC m=+8599.118026182 LastTransitionTime:2022-04-09 11:38:08 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-04-09T13:59:41.986 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:59:41.982619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:59:41.998 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:59:41.995325 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:59:52.222 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:59:52.218138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T13:59:52.234 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 13:59:52.230863 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:00:02.455 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:00:02.451445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:00:02.466 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:00:02.462759 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:00:12.720 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:00:12.717248 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:00:12.728 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:00:12.724992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:00:22.944 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:00:22.941165 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:00:23.008 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:00:23.004645 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:00:33.228 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:00:33.224490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:00:33.238 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:00:33.234958 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:00:43.459 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:00:43.455556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:00:43.469 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:00:43.466700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:00:53.689 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:00:53.685487 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:00:53.700 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:00:53.697078 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:01:03.923 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:01:03.920019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:01:03.936 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:01:03.932886 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:01:14.163 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:01:14.159384 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:01:14.175 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:01:14.171798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:01:24.396 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:01:24.392920 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:01:24.414 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:01:24.409349 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:01:34.637 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:01:34.633933 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:01:34.649 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:01:34.646486 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:01:44.889 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:01:44.885322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:01:44.902 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:01:44.899235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:01:55.123 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:01:55.119673 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:01:55.135 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:01:55.131598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:02:05.357 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:02:05.353218 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:02:05.370 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:02:05.367211 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:02:15.587 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:02:15.583596 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:02:15.597 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:02:15.594697 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:02:25.818 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:02:25.815046 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:02:25.828 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:02:25.824569 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:02:36.054 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:02:36.050884 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:02:36.066 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:02:36.063284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:02:46.292 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:02:46.288424 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:02:46.303 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:02:46.299725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:02:56.532 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:02:56.528661 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:02:56.543 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:02:56.540469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:03:06.765 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:03:06.762127 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:03:06.777 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:03:06.774616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:03:17.001 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:03:16.997944 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:03:17.013 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:03:17.010536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:03:27.245 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:03:27.241687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:03:27.259 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:03:27.253752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:03:37.484 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:03:37.480357 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:03:37.498 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:03:37.495930 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:03:47.719 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:03:47.715784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:03:47.732 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:03:47.728927 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:03:57.975 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:03:57.972148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:03:58.009 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:03:58.006725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:04:08.253 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:04:08.250367 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:04:08.333 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:04:08.330461 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:04:18.553 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:04:18.550003 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:04:18.566 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:04:18.563085 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:04:28.787 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:04:28.783732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:04:28.799 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:04:28.796413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:04:39.018 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:04:39.014395 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:04:39.027 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:04:39.024287 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:04:49.246 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:04:49.243294 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:04:49.258 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:04:49.255209 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:04:59.481 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:04:59.478146 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:04:59.495 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:04:59.492378 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:05:09.716 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:05:09.712823 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:05:09.727 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:05:09.724402 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:05:19.951 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:05:19.947775 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:05:19.967 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:05:19.962962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:05:30.190 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:05:30.184890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:05:30.200 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:05:30.197539 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:05:40.431 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:05:40.428355 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:05:40.445 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:05:40.442244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:05:50.667 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:05:50.663926 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:05:50.677 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:05:50.672645 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:06:00.901 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:06:00.898628 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:06:01.017 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:06:01.014407 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:06:11.236 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:06:11.232641 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:06:11.249 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:06:11.246219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:06:21.469 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:06:21.466244 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:06:21.480 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:06:21.477224 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:06:31.706 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:06:31.703320 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:06:31.716 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:06:31.713852 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:06:41.937 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:06:41.934010 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:06:41.947 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:06:41.944932 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:06:52.167 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:06:52.164518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:06:52.181 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:06:52.178056 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:07:02.408 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:07:02.405243 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:07:02.421 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:07:02.417924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:07:12.643 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:07:12.640236 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:07:12.650 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:07:12.647503 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:07:22.873 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:07:22.870608 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:07:22.887 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:07:22.883712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:07:33.111 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:07:33.107588 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:07:33.123 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:07:33.120247 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:07:43.346 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:07:43.342833 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:07:43.357 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:07:43.355068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:07:53.587 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:07:53.584236 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:07:53.601 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:07:53.598137 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:08:03.839 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:08:03.835738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:08:03.873 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:08:03.870735 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:08:14.097 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:08:14.088014 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:08:14.152 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:08:14.148851 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:08:24.371 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:08:24.367474 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:08:24.382 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:08:24.379687 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:08:34.597 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:08:34.594695 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:08:34.607 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:08:34.603813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:08:44.826 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:08:44.822925 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:08:44.838 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:08:44.834356 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:08:55.057 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:08:55.053905 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:08:55.070 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:08:55.067191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:09:05.293 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:09:05.290017 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:09:05.302 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:09:05.299455 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:09:15.526 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:09:15.523303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:09:15.539 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:09:15.536297 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:09:25.764 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:09:25.760679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:09:25.778 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:09:25.774376 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:09:36.002 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:09:35.998720 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:09:36.014 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:09:36.010856 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:09:46.239 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:09:46.236477 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:09:46.251 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:09:46.246915 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:09:56.476 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:09:56.470387 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:09:56.489 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:09:56.484324 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:10:06.712 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:10:06.708836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:10:06.747 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:10:06.744019 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:10:16.976 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:10:16.972836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:10:17.040 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:10:16.985514 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:10:27.228 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:10:27.225588 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:10:27.241 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:10:27.238807 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:10:37.467 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:10:37.463833 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:10:37.478 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:10:37.475713 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:10:47.705 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:10:47.702521 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:10:47.757 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:10:47.714557 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:10:57.941 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:10:57.938094 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:10:57.952 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:10:57.949615 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:11:08.178 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:11:08.175170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:11:08.190 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:11:08.187682 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:11:18.416 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:11:18.413081 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:11:18.430 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:11:18.427302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:11:28.652 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:11:28.649356 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:11:28.660 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:11:28.657898 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:11:38.885 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:11:38.881626 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:11:38.897 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:11:38.894791 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:11:49.121 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:11:49.118092 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:11:49.134 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:11:49.131228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:11:59.355 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:11:59.351880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:11:59.373 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:11:59.369514 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:12:09.591 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:12:09.588172 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:12:09.661 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:12:09.599640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:12:09.661 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:12:09.599667 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:12:19.821 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:12:19.817482 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:12:19.868 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:12:19.865143 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:12:30.089 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:12:30.085644 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:12:30.133 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:12:30.129927 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:12:40.386 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:12:40.382249 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:12:40.399 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:12:40.396844 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:12:50.637 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:12:50.633611 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:12:50.649 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:12:50.646153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:13:00.870 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:13:00.867273 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:13:00.881 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:13:00.878482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:13:11.098 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:13:11.094409 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:13:11.111 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:13:11.108104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:13:21.334 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:13:21.330799 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:13:21.346 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:13:21.343229 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:13:31.569 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:13:31.565824 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:13:31.580 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:13:31.577143 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:13:41.805 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:13:41.801608 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:13:41.815 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:13:41.812520 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:13:52.039 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:13:52.035815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:13:52.282 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:13:52.048013 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:14:02.279 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:14:02.276158 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:14:02.291 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:14:02.288215 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:14:12.518 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:14:12.515168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:14:12.528 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:14:12.525283 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:14:22.748 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:14:22.744842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:14:22.802 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:14:22.756951 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:14:30.165 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:14:30.162438 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T14:14:30.781 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:14:30.778271 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T14:14:30.767+0000 7fd9b58ba340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T14:14:30.782 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T14:14:30.767+0000 7fd9b58ba340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T14:14:30.782 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T14:14:30.767+0000 7fd9b58ba340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T14:14:30.782 INFO:tasks.rook.operator.smithi119.stdout:couldn't init storage provider. : exit status 5 2022-04-09T14:14:30.783 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:14:30.778300 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T14:14:30.767+0000 7fd9b58ba340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T14:14:30.783 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T14:14:30.767+0000 7fd9b58ba340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T14:14:30.783 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T14:14:30.767+0000 7fd9b58ba340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T14:14:30.783 INFO:tasks.rook.operator.smithi119.stdout:couldn't init storage provider. : exit status 5 2022-04-09T14:14:32.984 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:14:32.981269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:14:32.999 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:14:32.995266 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:14:43.222 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:14:43.218879 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:14:43.232 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:14:43.229622 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:14:53.460 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:14:53.456901 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:14:53.470 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:14:53.467023 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:15:03.732 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:15:03.728724 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:15:03.743 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:15:03.740161 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:15:13.991 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:15:13.988061 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:15:14.003 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:15:14.000169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:15:24.224 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:15:24.221082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:15:24.238 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:15:24.235121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:15:34.459 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:15:34.456161 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:15:34.471 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:15:34.468190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:15:44.691 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:15:44.687273 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:15:44.705 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:15:44.702383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:15:54.927 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:15:54.923718 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:15:54.940 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:15:54.937191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:16:05.190 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:16:05.186830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:16:05.201 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:16:05.198380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:16:15.423 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:16:15.419590 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:16:15.445 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:16:15.442244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:16:25.663 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:16:25.660344 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:16:25.704 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:16:25.701666 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:16:35.919 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:16:35.915562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:16:35.932 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:16:35.929511 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:16:46.149 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:16:46.145850 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:16:46.159 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:16:46.156629 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:16:56.380 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:16:56.376754 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:16:56.397 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:16:56.394073 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:17:06.646 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:17:06.643283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:17:06.656 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:17:06.653513 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:17:16.875 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:17:16.872288 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:17:16.885 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:17:16.882602 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:17:27.107 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:17:27.103494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:17:27.120 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:17:27.116970 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:17:37.341 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:17:37.338063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:17:37.354 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:17:37.350812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:17:47.528 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:17:47.524431 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-04-09 14:17:47.51817591 +0000 UTC m=+9694.879075484 LastTransitionTime:2022-04-09 11:38:08 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-04-09T14:17:47.573 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:17:47.569720 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:17:47.583 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:17:47.580918 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:17:57.802 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:17:57.798929 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:17:57.816 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:17:57.813412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:18:08.044 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:18:08.040733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:18:08.052 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:18:08.048970 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:18:18.271 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:18:18.267757 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:18:18.285 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:18:18.281554 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:18:28.513 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:18:28.503360 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:18:28.519 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:18:28.516991 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:18:38.745 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:18:38.741320 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:18:38.757 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:18:38.754314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:18:48.977 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:18:48.974296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:18:48.989 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:18:48.986433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:18:59.220 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:18:59.217455 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:18:59.233 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:18:59.230023 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:19:09.494 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:19:09.490352 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:19:09.504 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:19:09.500885 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:19:19.732 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:19:19.728893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:19:19.743 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:19:19.740516 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:19:29.963 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:19:29.960015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:19:29.975 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:19:29.972782 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:19:40.200 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:19:40.196717 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:19:40.213 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:19:40.209728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:19:50.434 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:19:50.430937 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:19:50.447 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:19:50.444153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:20:00.665 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:20:00.661172 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:20:00.673 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:20:00.670110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:20:10.900 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:20:10.896492 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:20:10.911 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:20:10.908699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:20:21.159 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:20:21.156083 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:20:21.172 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:20:21.168896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:20:31.391 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:20:31.387721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:20:31.402 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:20:31.399372 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:20:41.628 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:20:41.624703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:20:41.641 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:20:41.638156 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:20:51.859 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:20:51.856120 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:20:51.869 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:20:51.866712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:21:02.084 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:21:02.081226 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:21:02.096 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:21:02.093602 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:21:12.317 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:21:12.313676 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:21:12.328 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:21:12.325470 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:21:22.547 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:21:22.544123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:21:22.557 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:21:22.554147 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:21:32.779 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:21:32.775950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:21:32.791 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:21:32.788780 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:21:43.009 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:21:43.006189 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:21:43.024 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:21:43.021718 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:21:53.245 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:21:53.242195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:21:53.258 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:21:53.255002 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:22:03.473 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:22:03.469939 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:22:03.487 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:22:03.482859 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:22:13.706 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:22:13.703447 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:22:13.718 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:22:13.715019 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:22:23.939 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:22:23.935381 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:22:23.949 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:22:23.945864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:22:34.171 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:22:34.167388 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:22:34.181 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:22:34.178219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:22:44.405 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:22:44.401449 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:22:44.418 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:22:44.415285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:22:54.639 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:22:54.635657 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:22:54.649 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:22:54.645977 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:23:04.871 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:23:04.867468 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:23:04.880 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:23:04.877421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:23:15.103 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:23:15.099776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:23:15.115 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:23:15.112717 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:23:25.342 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:23:25.338516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:23:25.352 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:23:25.349256 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:23:35.578 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:23:35.574176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:23:35.591 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:23:35.588145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:23:45.815 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:23:45.811525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:23:45.828 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:23:45.824972 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:23:56.072 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:23:56.068383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:23:56.079 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:23:56.076692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:24:06.324 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:24:06.320559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:24:06.336 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:24:06.333577 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:24:16.576 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:24:16.573054 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:24:16.604 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:24:16.600737 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:24:26.827 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:24:26.824193 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:24:26.841 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:24:26.837443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:24:37.089 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:24:37.086189 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:24:37.099 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:24:37.095818 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:24:47.320 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:24:47.316620 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:24:47.333 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:24:47.330142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:24:57.553 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:24:57.549969 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:24:57.563 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:24:57.560252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:25:07.787 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:25:07.783752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:25:07.799 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:25:07.795974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:25:18.018 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:25:18.014317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:25:18.032 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:25:18.028896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:25:28.289 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:25:28.285079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:25:28.300 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:25:28.297273 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:25:38.523 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:25:38.519524 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:25:38.536 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:25:38.532881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:25:48.759 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:25:48.755992 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:25:48.771 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:25:48.768210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:25:59.055 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:25:59.052072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:25:59.066 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:25:59.062783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:26:09.288 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:26:09.284696 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:26:09.301 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:26:09.298086 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:26:19.524 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:26:19.520362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:26:19.537 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:26:19.533814 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:26:29.757 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:26:29.753253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:26:29.770 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:26:29.766671 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:26:40.003 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:26:39.999713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:26:40.013 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:26:40.010099 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:26:50.233 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:26:50.229700 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:26:50.246 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:26:50.243336 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:27:00.474 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:27:00.470646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:27:00.487 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:27:00.483768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:27:10.710 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:27:10.706778 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:27:10.733 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:27:10.725919 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:27:20.952 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:27:20.949104 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:27:20.997 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:27:20.993275 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:27:31.328 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:27:31.324991 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:27:31.345 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:27:31.337540 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:27:41.565 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:27:41.561451 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:27:41.575 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:27:41.572060 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:27:51.800 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:27:51.796681 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:27:51.819 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:27:51.811402 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:28:02.056 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:28:02.052547 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:28:02.133 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:28:02.129610 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:28:12.359 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:28:12.355053 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:28:12.366 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:28:12.363172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:28:22.594 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:28:22.590632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:28:22.605 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:28:22.601317 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:28:32.867 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:28:32.864079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:28:32.879 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:28:32.875975 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:28:43.135 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:28:43.131202 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:28:43.147 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:28:43.144410 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:28:53.371 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:28:53.366970 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:28:53.382 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:28:53.379454 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:29:03.599 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:29:03.595609 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:29:03.613 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:29:03.609851 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:29:13.835 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:29:13.831448 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:29:13.871 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:29:13.868069 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:29:24.099 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:29:24.095784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:29:24.110 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:29:24.107465 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:29:34.332 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:29:34.328457 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:29:34.344 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:29:34.340966 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:29:44.572 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:29:44.564518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:29:44.579 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:29:44.574938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:29:54.799 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:29:54.796052 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:29:54.810 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:29:54.807573 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:30:05.036 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:30:05.032588 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:30:05.051 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:30:05.047445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:30:15.275 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:30:15.271399 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:30:15.334 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:30:15.330805 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:30:25.555 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:30:25.551470 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:30:25.567 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:30:25.563433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:30:35.797 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:30:35.789453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:30:36.093 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:30:35.809623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:30:46.036 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:30:46.033209 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:30:46.049 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:30:46.044754 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:30:56.272 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:30:56.268411 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:30:56.284 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:30:56.280681 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:31:06.503 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:31:06.499166 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:31:06.515 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:31:06.511479 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:31:10.793 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:31:10.789006 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T14:31:11.399 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:31:11.392294 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T14:31:11.383+0000 7efda5e29340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T14:31:11.400 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T14:31:11.383+0000 7efda5e29340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T14:31:11.400 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T14:31:11.383+0000 7efda5e29340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T14:31:11.400 INFO:tasks.rook.operator.smithi119.stdout:couldn't init storage provider. : exit status 5 2022-04-09T14:31:11.400 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:31:11.392320 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T14:31:11.383+0000 7efda5e29340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T14:31:11.401 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T14:31:11.383+0000 7efda5e29340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T14:31:11.401 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T14:31:11.383+0000 7efda5e29340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T14:31:11.401 INFO:tasks.rook.operator.smithi119.stdout:couldn't init storage provider. : exit status 5 2022-04-09T14:31:16.738 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:31:16.734801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:31:16.750 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:31:16.747144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:31:26.967 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:31:26.963524 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:31:26.979 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:31:26.976350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:31:37.200 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:31:37.196594 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:31:37.213 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:31:37.208370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:31:47.441 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:31:47.437334 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:31:47.454 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:31:47.450658 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:31:57.680 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:31:57.676252 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:31:57.691 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:31:57.687544 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:32:07.914 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:32:07.910668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:32:07.926 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:32:07.922939 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:32:18.151 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:32:18.147634 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:32:18.166 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:32:18.162431 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:32:28.403 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:32:28.400039 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:32:28.415 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:32:28.412124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:32:38.680 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:32:38.676507 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:32:38.691 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:32:38.688469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:32:48.929 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:32:48.925958 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:32:48.941 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:32:48.937743 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:32:59.190 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:32:59.187210 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:32:59.204 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:32:59.200552 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:33:09.425 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:33:09.420917 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:33:09.436 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:33:09.433501 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:33:19.657 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:33:19.653071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:33:19.673 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:33:19.669446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:33:29.898 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:33:29.894634 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:33:29.910 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:33:29.906763 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:33:40.136 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:33:40.132461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:33:40.149 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:33:40.145918 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:33:50.376 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:33:50.372256 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:33:50.389 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:33:50.385505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:34:00.618 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:34:00.614842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:34:00.632 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:34:00.628744 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:34:10.859 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:34:10.855243 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:34:10.872 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:34:10.868229 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:34:21.095 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:34:21.091697 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:34:21.106 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:34:21.102706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:34:31.330 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:34:31.326523 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:34:31.341 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:34:31.338279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:34:41.562 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:34:41.557974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:34:41.618 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:34:41.614299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:34:51.844 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:34:51.840674 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:34:51.855 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:34:51.852489 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:35:02.070 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:35:02.066804 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:35:02.082 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:35:02.079160 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:35:02.378 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:35:02.374294 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-04-09 14:35:02.364731581 +0000 UTC m=+10729.725631155 LastTransitionTime:2022-04-09 11:38:08 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-04-09T14:35:12.301 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:35:12.298047 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:35:12.314 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:35:12.308613 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:35:22.533 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:35:22.528923 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:35:22.546 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:35:22.542952 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:35:32.763 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:35:32.759938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:35:32.777 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:35:32.773706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:35:42.998 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:35:42.994395 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:35:43.010 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:35:43.006675 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:35:53.235 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:35:53.231120 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:35:53.246 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:35:53.243308 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:36:03.469 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:36:03.465623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:36:03.480 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:36:03.476948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:36:13.710 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:36:13.706932 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:36:13.722 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:36:13.718662 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:36:23.947 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:36:23.943233 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:36:23.960 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:36:23.955077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:36:34.235 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:36:34.231045 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:36:34.244 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:36:34.240476 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:36:44.463 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:36:44.458752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:36:44.475 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:36:44.471698 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:36:54.693 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:36:54.689822 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:36:54.705 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:36:54.700851 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:37:04.930 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:37:04.926368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:37:04.982 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:37:04.937603 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:37:15.167 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:37:15.163117 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:37:15.179 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:37:15.175114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:37:25.446 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:37:25.442151 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:37:25.455 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:37:25.451831 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:37:35.697 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:37:35.693629 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:37:35.710 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:37:35.705537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:37:45.924 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:37:45.920669 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:37:45.935 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:37:45.932047 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:37:56.160 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:37:56.156176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:37:56.172 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:37:56.168726 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:38:06.396 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:38:06.391965 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:38:06.407 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:38:06.403842 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:38:16.639 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:38:16.634403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:38:16.651 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:38:16.647159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:38:26.874 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:38:26.870828 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:38:26.888 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:38:26.884679 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:38:37.113 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:38:37.109315 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:38:37.126 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:38:37.122826 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:38:47.352 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:38:47.347991 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:38:47.363 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:38:47.359443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:38:57.589 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:38:57.585356 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:38:57.604 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:38:57.599689 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:39:07.832 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:39:07.828107 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:39:07.845 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:39:07.841333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:39:18.074 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:39:18.070542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:39:18.084 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:39:18.080700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:39:28.315 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:39:28.310988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:39:28.323 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:39:28.319488 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:39:38.549 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:39:38.544671 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:39:38.560 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:39:38.556974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:39:48.815 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:39:48.811272 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:39:48.823 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:39:48.819974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:39:59.048 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:39:59.044400 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:39:59.057 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:39:59.053485 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:40:09.280 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:40:09.276411 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:40:09.291 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:40:09.287581 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:40:19.517 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:40:19.512896 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:40:19.530 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:40:19.526945 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:40:29.756 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:40:29.751767 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:40:29.769 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:40:29.765203 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:40:39.985 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:40:39.981553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:40:40.029 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:40:40.026155 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:40:50.262 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:40:50.258039 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:40:50.279 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:40:50.275273 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:41:00.554 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:41:00.550138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:41:00.565 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:41:00.561385 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:41:10.783 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:41:10.779246 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:41:10.791 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:41:10.787487 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:41:21.014 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:41:21.010091 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:41:21.030 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:41:21.023332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:41:31.249 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:41:31.244475 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:41:31.261 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:41:31.256327 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:41:41.527 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:41:41.523708 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:41:41.540 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:41:41.536167 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:41:51.760 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:41:51.756368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:41:51.772 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:41:51.768927 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:42:01.992 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:42:01.988908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:42:02.001 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:42:01.997354 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:42:12.248 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:42:12.244552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:42:12.261 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:42:12.257506 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:42:22.486 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:42:22.482020 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:42:22.498 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:42:22.495130 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:42:32.718 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:42:32.714111 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:42:32.730 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:42:32.726815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:42:42.957 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:42:42.952958 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:42:43.003 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:42:42.999086 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:42:53.227 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:42:53.223484 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:42:53.239 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:42:53.235249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:43:03.461 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:43:03.457596 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:43:03.473 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:43:03.470208 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:43:13.697 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:43:13.692834 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:43:13.709 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:43:13.705909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:43:23.931 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:43:23.927770 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:43:23.943 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:43:23.939598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:43:34.168 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:43:34.164092 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:43:34.180 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:43:34.177188 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:43:44.416 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:43:44.411919 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:43:44.431 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:43:44.427196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:43:54.651 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:43:54.647581 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:43:54.663 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:43:54.659886 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:44:04.898 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:44:04.894204 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:44:04.910 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:44:04.905928 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:44:15.129 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:44:15.125197 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:44:15.142 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:44:15.138495 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:44:25.369 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:44:25.365271 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:44:25.381 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:44:25.377462 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:44:35.605 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:44:35.596985 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:44:35.638 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:44:35.633649 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:44:45.857 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:44:45.853185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:44:45.868 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:44:45.864962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:44:56.091 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:44:56.086820 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:44:56.109 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:44:56.101213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:45:06.372 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:45:06.368820 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:45:06.382 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:45:06.378261 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:45:16.607 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:45:16.603627 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:45:16.620 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:45:16.616527 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:45:26.840 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:45:26.836689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:45:26.853 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:45:26.849625 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:45:37.075 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:45:37.071257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:45:37.086 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:45:37.082734 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:45:47.305 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:45:47.301101 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:45:47.318 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:45:47.314695 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:45:57.538 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:45:57.534380 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:45:57.582 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:45:57.578899 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:46:07.810 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:46:07.806138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:46:07.822 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:46:07.818544 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:46:18.048 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:46:18.044591 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:46:18.060 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:46:18.057221 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:46:28.286 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:46:28.282007 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:46:28.296 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:46:28.292679 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:46:38.521 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:46:38.517150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:46:38.533 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:46:38.529364 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:46:48.757 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:46:48.753304 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:46:48.769 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:46:48.762620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:46:59.004 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:46:59.000123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:46:59.015 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:46:59.012077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:47:09.238 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:47:09.234755 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:47:09.250 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:47:09.246771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:47:19.473 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:47:19.469493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:47:19.486 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:47:19.482758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:47:29.710 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:47:29.706105 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:47:29.721 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:47:29.718210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:47:39.946 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:47:39.941971 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:47:39.957 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:47:39.954164 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:47:50.181 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:47:50.177444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:47:50.192 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:47:50.189241 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:47:51.405 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:47:51.401424 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T14:47:52.084 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:47:52.080282 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T14:47:52.070+0000 7fe2b00d3340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T14:47:52.085 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T14:47:52.070+0000 7fe2b00d3340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T14:47:52.085 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T14:47:52.070+0000 7fe2b00d3340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T14:47:52.086 INFO:tasks.rook.operator.smithi119.stdout:couldn't init storage provider. : exit status 5 2022-04-09T14:47:52.086 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:47:52.080307 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T14:47:52.070+0000 7fe2b00d3340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T14:47:52.086 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T14:47:52.070+0000 7fe2b00d3340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T14:47:52.087 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T14:47:52.070+0000 7fe2b00d3340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T14:47:52.087 INFO:tasks.rook.operator.smithi119.stdout:couldn't init storage provider. : exit status 5 2022-04-09T14:48:00.415 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:48:00.411936 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:48:00.426 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:48:00.422864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:48:10.649 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:48:10.645970 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:48:10.658 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:48:10.655388 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:48:20.886 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:48:20.882044 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:48:20.898 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:48:20.894854 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:48:31.120 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:48:31.115208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:48:31.131 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:48:31.127162 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:48:41.347 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:48:41.343343 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:48:41.369 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:48:41.365907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:48:51.605 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:48:51.601830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:48:51.618 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:48:51.614873 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:49:01.837 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:49:01.833748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:49:01.856 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:49:01.853077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:49:12.097 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:49:12.093506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:49:12.111 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:49:12.107918 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:49:22.328 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:49:22.323932 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:49:22.340 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:49:22.335507 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:49:32.556 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:49:32.551978 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:49:32.567 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:49:32.562971 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:49:42.786 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:49:42.782527 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:49:42.818 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:49:42.814455 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:49:53.050 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:49:53.037449 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:49:53.057 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:49:53.050459 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:50:03.275 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:50:03.271863 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:50:03.287 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:50:03.283733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:50:13.511 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:50:13.507365 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:50:13.525 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:50:13.522079 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:50:23.747 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:50:23.743688 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:50:23.762 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:50:23.758917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:50:33.985 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:50:33.981621 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:50:34.000 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:50:33.996319 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:50:44.223 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:50:44.218919 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:50:44.233 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:50:44.229861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:50:54.456 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:50:54.452242 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:50:54.467 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:50:54.463697 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:51:04.706 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:51:04.701908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:51:04.719 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:51:04.714039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:51:14.942 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:51:14.937921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:51:14.955 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:51:14.950380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:51:25.209 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:51:25.204975 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:51:25.221 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:51:25.218012 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:51:35.473 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:51:35.469533 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:51:35.486 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:51:35.483078 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:51:45.713 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:51:45.709430 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:51:45.724 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:51:45.721420 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:51:55.947 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:51:55.943017 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:51:55.957 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:51:55.954245 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:52:06.184 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:52:06.180470 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:52:06.197 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:52:06.193706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:52:16.417 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:52:16.413950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:52:16.431 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:52:16.427879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:52:26.673 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:52:26.669628 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:52:26.688 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:52:26.684268 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:52:36.938 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:52:36.934100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:52:36.947 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:52:36.944148 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:52:47.262 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:52:47.258671 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:52:47.275 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:52:47.271864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:52:57.520 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:52:57.516923 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:52:57.567 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:52:57.562370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:53:07.784 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:53:07.780738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:53:07.797 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:53:07.793692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:53:17.981 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:53:17.977682 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-04-09 14:53:17.970379614 +0000 UTC m=+11825.331279147 LastTransitionTime:2022-04-09 11:38:08 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-04-09T14:53:18.034 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:53:18.030906 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:53:18.142 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:53:18.138217 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:53:28.442 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:53:28.438732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:53:28.456 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:53:28.452556 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:53:38.674 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:53:38.670493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:53:38.683 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:53:38.679917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:53:48.943 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:53:48.939327 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:53:48.958 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:53:48.954515 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:53:59.175 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:53:59.171471 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:53:59.188 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:53:59.184605 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:54:09.408 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:54:09.404158 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:54:09.421 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:54:09.418078 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:54:19.655 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:54:19.651359 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:54:19.669 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:54:19.666112 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:54:29.890 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:54:29.886152 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:54:29.904 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:54:29.900611 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:54:40.129 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:54:40.125068 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:54:40.140 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:54:40.136636 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:54:50.363 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:54:50.359598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:54:50.397 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:54:50.393784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:55:00.624 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:55:00.620098 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:55:00.636 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:55:00.633037 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:55:10.858 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:55:10.854654 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:55:10.869 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:55:10.865541 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:55:21.093 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:55:21.088997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:55:21.106 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:55:21.103099 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:55:31.377 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:55:31.373654 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:55:31.391 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:55:31.387133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:55:41.621 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:55:41.617736 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:55:41.630 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:55:41.626838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:55:51.845 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:55:51.841850 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:55:51.857 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:55:51.853973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:56:02.081 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:56:02.077040 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:56:02.124 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:56:02.121329 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:56:12.372 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:56:12.368168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:56:12.383 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:56:12.380109 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:56:22.609 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:56:22.605380 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:56:22.697 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:56:22.693980 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:56:32.940 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:56:32.936238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:56:32.971 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:56:32.967597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:56:43.193 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:56:43.189442 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:56:43.201 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:56:43.198335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:56:53.444 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:56:53.440401 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:56:53.456 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:56:53.452749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:57:03.674 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:57:03.670720 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:57:03.686 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:57:03.683007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:57:13.933 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:57:13.929410 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:57:13.943 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:57:13.939838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:57:24.163 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:57:24.159481 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:57:24.175 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:57:24.171506 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:57:34.394 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:57:34.390078 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:57:34.408 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:57:34.404772 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:57:44.630 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:57:44.627083 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:57:44.641 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:57:44.638319 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:57:54.867 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:57:54.863622 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:57:54.880 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:57:54.876809 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:58:05.105 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:58:05.100953 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:58:05.117 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:58:05.113681 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:58:15.342 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:58:15.338090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:58:15.354 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:58:15.350941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:58:25.572 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:58:25.567997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:58:25.584 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:58:25.581104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:58:35.807 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:58:35.803091 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:58:35.820 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:58:35.816775 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:58:46.047 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:58:46.043437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:58:46.064 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:58:46.060470 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:58:56.287 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:58:56.283950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:58:56.306 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:58:56.295598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:59:06.521 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:59:06.517621 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:59:06.534 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:59:06.530832 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:59:16.756 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:59:16.752436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:59:16.767 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:59:16.763994 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:59:26.987 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:59:26.983925 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:59:27.045 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:59:27.041360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:59:37.263 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:59:37.259727 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:59:37.271 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:59:37.268178 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:59:47.494 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:59:47.490292 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:59:47.506 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:59:47.503062 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:59:57.726 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:59:57.722213 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T14:59:57.737 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 14:59:57.734427 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:00:07.967 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:00:07.963903 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:00:07.978 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:00:07.974630 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:00:18.205 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:00:18.200514 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:00:18.217 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:00:18.213765 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:00:28.469 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:00:28.465380 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:00:28.481 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:00:28.477641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:00:38.705 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:00:38.700789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:00:38.716 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:00:38.712639 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:00:48.936 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:00:48.932162 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:00:48.948 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:00:48.944381 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:00:59.166 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:00:59.162907 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:00:59.176 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:00:59.173040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:01:09.444 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:01:09.439816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:01:09.455 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:01:09.451761 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:01:19.681 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:01:19.678006 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:01:19.691 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:01:19.688385 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:01:29.911 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:01:29.907388 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:01:29.922 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:01:29.918794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:01:40.149 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:01:40.145402 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:01:40.161 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:01:40.157840 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:01:50.409 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:01:50.405538 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:01:50.420 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:01:50.416124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:02:00.639 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:02:00.635649 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:02:00.651 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:02:00.647315 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:02:10.868 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:02:10.864519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:02:10.880 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:02:10.877255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:02:21.099 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:02:21.095904 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:02:21.111 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:02:21.108120 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:02:31.334 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:02:31.330055 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:02:31.346 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:02:31.342250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:02:41.571 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:02:41.567866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:02:41.582 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:02:41.579398 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:02:51.805 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:02:51.800995 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:02:51.817 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:02:51.813648 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:03:02.038 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:03:02.034126 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:03:02.050 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:03:02.046394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:03:12.282 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:03:12.278076 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:03:12.294 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:03:12.290319 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:03:22.517 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:03:22.513064 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:03:22.530 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:03:22.526221 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:03:32.751 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:03:32.747750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:03:32.764 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:03:32.760887 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:03:42.989 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:03:42.985010 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:03:43.001 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:03:42.997769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:03:53.227 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:03:53.223271 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:03:53.242 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:03:53.238690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:04:03.459 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:04:03.455164 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:04:03.470 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:04:03.466429 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:04:13.704 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:04:13.700995 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:04:13.717 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:04:13.713333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:04:23.940 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:04:23.936837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:04:23.952 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:04:23.948847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:04:32.097 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:04:32.093893 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T15:04:32.706 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:04:32.702369 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T15:04:32.693+0000 7f91ba566340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T15:04:32.707 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T15:04:32.693+0000 7f91ba566340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T15:04:32.707 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T15:04:32.693+0000 7f91ba566340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T15:04:32.708 INFO:tasks.rook.operator.smithi119.stdout:couldn't init storage provider. : exit status 5 2022-04-09T15:04:32.708 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:04:32.702392 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T15:04:32.693+0000 7f91ba566340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T15:04:32.708 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T15:04:32.693+0000 7f91ba566340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T15:04:32.708 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T15:04:32.693+0000 7f91ba566340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T15:04:32.709 INFO:tasks.rook.operator.smithi119.stdout:couldn't init storage provider. : exit status 5 2022-04-09T15:04:34.173 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:04:34.169285 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:04:34.186 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:04:34.183233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:04:44.404 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:04:44.400085 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:04:44.416 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:04:44.410910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:04:54.650 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:04:54.646795 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:04:54.663 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:04:54.657775 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:05:04.890 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:05:04.886838 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:05:04.900 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:05:04.896272 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:05:15.148 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:05:15.144414 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:05:15.161 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:05:15.157478 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:05:25.381 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:05:25.377539 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:05:25.392 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:05:25.389200 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:05:35.614 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:05:35.609960 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:05:35.626 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:05:35.622425 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:05:45.846 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:05:45.842754 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:05:45.858 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:05:45.854794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:05:56.079 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:05:56.074778 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:05:56.091 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:05:56.087132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:06:06.312 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:06:06.308662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:06:06.325 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:06:06.321845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:06:16.549 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:06:16.545068 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:06:16.562 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:06:16.558612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:06:26.792 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:06:26.788658 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:06:26.804 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:06:26.801052 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:06:37.068 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:06:37.063200 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:06:37.080 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:06:37.076329 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:06:47.306 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:06:47.301761 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:06:47.318 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:06:47.314647 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:06:57.539 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:06:57.535761 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:06:57.553 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:06:57.549507 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:07:07.772 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:07:07.768736 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:07:07.785 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:07:07.782016 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:07:18.012 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:07:18.007302 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:07:18.025 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:07:18.020531 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:07:28.247 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:07:28.243078 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:07:28.257 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:07:28.254066 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:07:38.478 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:07:38.473794 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:07:38.487 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:07:38.483703 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:07:48.706 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:07:48.702798 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:07:48.717 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:07:48.713824 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:07:58.940 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:07:58.936316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:07:58.955 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:07:58.951877 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:08:09.179 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:08:09.175450 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:08:09.191 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:08:09.188273 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:08:19.418 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:08:19.414530 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:08:19.430 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:08:19.426541 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:08:29.707 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:08:29.702997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:08:29.718 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:08:29.714074 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:08:39.956 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:08:39.951842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:08:39.968 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:08:39.964695 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:08:50.187 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:08:50.183495 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:08:50.196 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:08:50.192061 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:09:00.417 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:09:00.413031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:09:00.431 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:09:00.427729 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:09:10.649 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:09:10.644688 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:09:10.661 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:09:10.657395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:09:20.880 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:09:20.876243 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:09:20.891 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:09:20.887833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:09:31.112 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:09:31.108155 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:09:31.122 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:09:31.118781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:09:41.344 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:09:41.339812 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:09:41.356 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:09:41.352950 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:09:51.583 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:09:51.578308 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:09:51.595 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:09:51.591663 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:10:01.818 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:10:01.814180 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:10:01.830 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:10:01.826199 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:10:12.051 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:10:12.045503 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:10:12.061 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:10:12.057359 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:10:22.318 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:10:22.313907 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:10:22.327 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:10:22.323240 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:10:32.540 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:10:32.536699 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:10:32.553 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:10:32.550213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:10:32.708 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:10:32.704673 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-04-09 15:10:32.696517678 +0000 UTC m=+12860.057417198 LastTransitionTime:2022-04-09 11:38:08 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-04-09T15:10:42.785 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:10:42.781095 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:10:42.796 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:10:42.792384 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:10:53.025 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:10:53.020867 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:10:53.036 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:10:53.032553 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:11:03.258 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:11:03.253981 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:11:03.269 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:11:03.266117 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:11:13.491 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:11:13.487223 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:11:13.503 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:11:13.499544 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:11:23.728 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:11:23.723810 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:11:23.741 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:11:23.738063 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:11:33.964 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:11:33.959924 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:11:33.976 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:11:33.972736 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:11:44.195 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:11:44.190861 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:11:44.207 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:11:44.203471 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:11:54.431 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:11:54.427026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:11:54.442 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:11:54.438440 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:12:04.668 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:12:04.663727 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:12:04.680 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:12:04.676546 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:12:14.901 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:12:14.897233 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:12:14.910 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:12:14.906862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:12:14.911 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:12:14.906905 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:12:25.138 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:12:25.133154 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:12:25.149 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:12:25.145714 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:12:35.370 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:12:35.366124 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:12:35.382 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:12:35.378258 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:12:45.650 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:12:45.644955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:12:45.658 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:12:45.654107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:12:55.888 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:12:55.883925 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:12:55.898 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:12:55.894501 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:13:06.119 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:13:06.114949 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:13:06.128 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:13:06.124407 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:13:16.377 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:13:16.372899 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:13:16.387 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:13:16.383790 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:13:26.606 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:13:26.602275 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:13:26.626 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:13:26.622424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:13:36.862 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:13:36.856948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:13:36.872 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:13:36.868973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:13:47.093 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:13:47.088830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:13:47.104 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:13:47.100264 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:13:57.327 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:13:57.323124 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:13:57.337 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:13:57.333427 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:14:07.553 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:14:07.548394 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:14:07.568 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:14:07.564272 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:14:17.792 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:14:17.788224 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:14:17.805 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:14:17.801019 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:14:28.025 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:14:28.021160 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:14:28.038 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:14:28.033111 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:14:38.255 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:14:38.251372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:14:38.267 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:14:38.264159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:14:48.489 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:14:48.485541 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:14:48.503 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:14:48.499054 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:14:58.723 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:14:58.718997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:14:58.734 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:14:58.730380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:15:08.956 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:15:08.952030 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:15:08.968 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:15:08.964830 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:15:19.194 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:15:19.190791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:15:19.206 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:15:19.202350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:15:29.438 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:15:29.433851 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:15:29.448 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:15:29.445200 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:15:39.671 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:15:39.667285 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:15:39.684 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:15:39.680538 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:15:49.908 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:15:49.904938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:15:49.921 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:15:49.917777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:16:00.184 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:16:00.180269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:16:00.196 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:16:00.192938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:16:10.423 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:16:10.419153 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:16:10.436 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:16:10.432211 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:16:20.667 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:16:20.661693 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:16:20.677 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:16:20.673961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:16:30.899 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:16:30.895842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:16:30.909 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:16:30.905549 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:16:41.131 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:16:41.127326 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:16:41.143 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:16:41.139349 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:16:51.382 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:16:51.378642 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:16:51.395 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:16:51.391977 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:17:01.613 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:17:01.609283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:17:01.621 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:17:01.618165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:17:11.887 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:17:11.883927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:17:11.899 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:17:11.896494 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:17:22.203 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:17:22.199412 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:17:22.238 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:17:22.235086 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:17:32.456 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:17:32.452414 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:17:32.464 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:17:32.460956 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:17:42.682 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:17:42.677748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:17:42.693 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:17:42.689842 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:17:52.931 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:17:52.927024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:17:52.960 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:17:52.956213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:18:03.179 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:18:03.176094 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:18:03.190 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:18:03.187544 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:18:13.413 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:18:13.408978 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:18:13.426 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:18:13.422287 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:18:23.649 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:18:23.645839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:18:23.662 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:18:23.659179 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:18:33.886 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:18:33.882603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:18:33.897 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:18:33.893305 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:18:44.119 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:18:44.115436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:18:44.132 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:18:44.128762 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:18:54.355 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:18:54.351111 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:18:54.367 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:18:54.363473 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:19:04.595 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:19:04.590707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:19:04.606 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:19:04.602358 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:19:14.826 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:19:14.822032 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:19:14.839 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:19:14.835892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:19:25.063 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:19:25.059500 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:19:25.078 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:19:25.074598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:19:35.304 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:19:35.301006 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:19:35.317 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:19:35.313892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:19:45.537 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:19:45.533404 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:19:45.548 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:19:45.544707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:19:55.774 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:19:55.770486 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:19:55.787 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:19:55.783793 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:20:06.006 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:20:06.002941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:20:06.017 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:20:06.013727 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:20:16.237 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:20:16.233585 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:20:16.247 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:20:16.244169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:20:26.478 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:20:26.474449 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:20:26.491 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:20:26.487888 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:20:36.753 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:20:36.750002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:20:36.766 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:20:36.763387 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:20:46.994 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:20:46.990668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:20:47.007 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:20:47.003573 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:20:57.242 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:20:57.238149 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:20:57.255 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:20:57.251398 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:21:07.490 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:21:07.486614 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:21:07.499 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:21:07.495510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:21:12.710 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:21:12.706673 I | op-mon: parsing mon endpoints: a=10.107.24.44:6789 2022-04-09T15:21:13.418 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:21:13.414094 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T15:21:13.404+0000 7fc41b6b3340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T15:21:13.419 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T15:21:13.404+0000 7fc41b6b3340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T15:21:13.419 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T15:21:13.404+0000 7fc41b6b3340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T15:21:13.419 INFO:tasks.rook.operator.smithi119.stdout:couldn't init storage provider. : exit status 5 2022-04-09T15:21:13.420 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:21:13.414126 I | op-k8sutil: Reporting Event rook-ceph:foo Warning:ReconcileFailed:failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-04-09T15:21:13.404+0000 7fc41b6b3340 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-04-09T15:21:13.420 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T15:21:13.404+0000 7fc41b6b3340 0 ERROR: failed to start notify service ((2) No such file or directory 2022-04-09T15:21:13.420 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09T15:21:13.404+0000 7fc41b6b3340 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-04-09T15:21:13.420 INFO:tasks.rook.operator.smithi119.stdout:couldn't init storage provider. : exit status 5 2022-04-09T15:21:17.718 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:21:17.714844 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:21:17.730 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:21:17.727019 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:21:27.987 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:21:27.983031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:21:27.999 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:21:27.996454 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:21:38.220 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:21:38.216921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:21:38.237 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:21:38.234064 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:21:48.456 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:21:48.453153 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:21:48.469 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:21:48.465675 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:21:58.689 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:21:58.685248 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:21:58.701 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:21:58.698350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:22:08.922 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:22:08.918478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:22:08.929 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:22:08.926232 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:22:19.150 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:22:19.146143 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:22:19.162 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:22:19.158873 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:22:29.383 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:22:29.379586 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:22:29.396 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:22:29.393370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:22:39.624 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:22:39.621013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:22:39.636 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:22:39.632483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:22:49.890 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:22:49.886959 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:22:49.899 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:22:49.896335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:23:00.117 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:23:00.114142 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:23:00.127 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:23:00.124402 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:23:10.351 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:23:10.348114 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:23:10.361 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:23:10.357652 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:23:20.582 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:23:20.578097 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:23:20.593 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:23:20.590155 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:23:30.816 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:23:30.812471 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:23:30.828 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:23:30.824600 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:23:41.052 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:23:41.048535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:23:41.065 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:23:41.062014 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:23:51.290 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:23:51.286358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:23:51.303 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:23:51.299398 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:24:01.525 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:24:01.521388 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:24:01.538 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:24:01.535239 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:24:11.762 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:24:11.758662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:24:11.773 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:24:11.769862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:24:22.011 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:24:22.007632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:24:22.025 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:24:22.021512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:24:32.249 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:24:32.245622 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:24:32.260 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:24:32.256574 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:24:42.482 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:24:42.478526 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:24:42.495 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:24:42.491877 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:24:52.722 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:24:52.719283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:24:52.763 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:24:52.759470 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:25:02.984 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:25:02.980634 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:25:03.016 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:25:03.012446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:25:13.274 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:25:13.270729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:25:13.283 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:25:13.279670 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:25:23.501 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:25:23.497829 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:25:23.514 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:25:23.510915 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:25:33.746 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:25:33.742362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:25:33.760 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:25:33.756720 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:25:44.003 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:25:43.999205 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:25:44.016 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:25:44.012074 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:25:54.242 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:25:54.238256 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:25:54.253 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:25:54.249626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:26:04.472 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:26:04.468763 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:26:04.487 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:26:04.483616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:26:14.709 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:26:14.705553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:26:14.722 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:26:14.718773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:26:24.945 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:26:24.941684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:26:24.958 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:26:24.954901 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:26:35.184 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:26:35.180797 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:26:35.194 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:26:35.191179 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:26:45.418 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:26:45.414194 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:26:45.430 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:26:45.427118 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:26:55.655 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:26:55.651902 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:26:55.667 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:26:55.663444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:27:05.897 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:27:05.893922 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:27:05.907 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:27:05.904248 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:27:16.143 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:27:16.139130 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:27:16.192 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:27:16.188380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:27:26.416 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:27:26.412074 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:27:26.425 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:27:26.422220 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:27:36.650 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:27:36.646155 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:27:36.659 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:27:36.656020 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:27:46.894 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:27:46.890559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:27:46.903 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:27:46.899970 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:27:57.120 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:27:57.116758 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:27:57.134 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:27:57.130520 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:28:07.356 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:28:07.352478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:28:07.367 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:28:07.364250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:28:17.593 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:28:17.589610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:28:17.607 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:28:17.604222 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:28:27.827 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:28:27.823386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:28:27.837 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:28:27.833753 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:28:38.061 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:28:38.057202 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:28:38.072 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:28:38.068670 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:28:48.297 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:28:48.293095 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:28:48.307 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:28:48.304118 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:28:48.449 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:28:48.445667 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-04-09 15:28:48.436079577 +0000 UTC m=+13955.796979142 LastTransitionTime:2022-04-09 11:38:08 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-04-09T15:28:58.534 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:28:58.530118 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:28:58.618 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:28:58.614044 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:29:08.836 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:29:08.832162 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:29:08.847 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:29:08.844353 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:29:19.080 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:29:19.076273 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:29:19.092 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:29:19.089070 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:29:29.307 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:29:29.303762 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:29:29.320 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:29:29.317415 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:29:39.541 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:29:39.537631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:29:39.584 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:29:39.580575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:29:49.800 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:29:49.796136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:29:49.814 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:29:49.810749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:30:00.033 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:30:00.029502 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:30:00.045 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:30:00.041251 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:30:10.267 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:30:10.263317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:30:10.280 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:30:10.276756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:30:20.500 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:30:20.496800 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:30:20.513 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:30:20.509584 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:30:30.738 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:30:30.734345 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:30:30.750 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:30:30.747090 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:30:40.974 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:30:40.970676 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:30:40.986 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:30:40.983256 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:30:51.211 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:30:51.207756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:30:51.222 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:30:51.219117 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:31:01.441 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:31:01.436684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:31:01.452 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:31:01.448866 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:31:11.673 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:31:11.668929 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:31:11.683 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:31:11.679928 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:31:21.899 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:31:21.895996 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:31:21.913 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:31:21.909397 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:31:32.136 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:31:32.132871 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:31:32.150 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:31:32.145276 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:31:42.371 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:31:42.367338 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:31:42.379 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:31:42.375877 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:31:52.637 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:31:52.633582 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:31:52.650 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:31:52.646884 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:32:02.870 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:32:02.866618 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:32:02.883 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:32:02.879897 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:32:13.104 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:32:13.099588 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:32:13.116 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:32:13.113150 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:32:23.337 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:32:23.332752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:32:23.349 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:32:23.346067 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:32:33.577 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:32:33.573254 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:32:33.591 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:32:33.587895 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:32:43.817 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:32:43.813043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:32:43.829 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:32:43.825376 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:32:54.050 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:32:54.046151 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:32:54.061 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:32:54.058072 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:33:04.284 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:33:04.280597 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:33:04.295 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:33:04.291518 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:33:14.555 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:33:14.550070 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:33:14.567 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:33:14.563677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:33:24.787 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:33:24.783356 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:33:24.796 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:33:24.793305 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:33:35.060 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:33:35.056156 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:33:35.137 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:33:35.133486 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:33:45.370 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:33:45.366408 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:33:45.383 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:33:45.379321 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:33:55.622 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:33:55.618380 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:33:55.631 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:33:55.627730 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:34:05.887 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:34:05.883923 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:34:05.903 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:34:05.899382 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:34:16.130 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:34:16.120974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:34:16.141 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:34:16.137893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:34:26.360 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:34:26.356379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:34:26.372 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:34:26.368766 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:34:36.594 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:34:36.589953 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:34:36.605 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:34:36.601200 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:34:46.833 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:34:46.828915 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:34:46.843 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:34:46.839393 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:34:57.065 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:34:57.061171 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:34:57.074 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:34:57.070441 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:35:07.299 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:35:07.295457 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:35:07.312 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:35:07.308080 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:35:17.542 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:35:17.535915 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:35:17.552 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:35:17.548365 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:35:27.776 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:35:27.772553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:35:27.806 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:35:27.796557 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:35:38.023 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:35:38.019691 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:35:38.034 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:35:38.031121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:35:48.256 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:35:48.252181 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:35:48.268 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:35:48.264252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:35:58.497 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:35:58.492750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:35:58.510 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:35:58.506777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:36:08.738 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:36:08.734277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T15:36:08.748 INFO:tasks.rook.operator.smithi119.stdout:2022-04-09 15:36:08.744934 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo] 2022-04-09T17:58:02.023 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-04-09T17:58:02.054 DEBUG:teuthology.task.console_log:Killing console logger for smithi119 2022-04-09T17:58:02.056 DEBUG:teuthology.exit:Finished running handlers